Skip to Content
  2. Mailing Lists
  3. Contributors
  4. Re: demo data and tests

Archives

Contributors

Re: demo data and tests

Re: demo data and tests

Re: demo data and tests

Hi

On my projects tests are run at 2 levels:
  • At development level: The developer tests the code he has written to ensure that it works as expected. For this, the developper writes unit tests that are run on an empty database with only this addon installed. This ensures that the code is correct and works properly in the context defined by the dependencies of the addon.
  • At integration level: The CI runs all the tests on a database with all the addons installed. This ensures that the code is correct and works properly in the context of the whole system.

To achieve this, we apply the following guidelines:
  • Tests don't rely on demo data; This eases the maintenance of the tests and ensures that the tests are not impacted by changes in the demo data. Moreover, it helps the developer to understand the context in which the tests are run.
  • Code that modifies / constrains an existing behavior must be disabled by default and must be enabled by a configuration parameter. This ensures that if you run the tests from other addons on a database where this addon is installed, the tests will not be impacted by this addon. It also ensures that if the tests are run at insall and another addon modifies the same behavior, the tests will not be impacted by this addon if it has been installed previously in the installation process. (The disabled behaviour is activated at test setup). All the disabled features are enabled in a top level addon which is excluded from the installation by the CI and installed only into the client database.
  •  Some specific test addons are developed for specific integration tests. These addons are useful to ensure that addons installed side by side are compatible even if they don't depend on each other. At test setup, we first activate the features of the addons that are activable by configuration parameter.
  • To avoid the need to use the "post_install' annotation, the CI runs the tests with pytest-odoo. The main difference compared to the way odoo runs the tests is that the registry is populated with all the addons installed into the database. pytest-odoo is a test runner only while when you run the tests with odoo, the tests are run along the installation/upgrade of the addons at the same time the registry is populated. To run tests with pytest-odoo, the first requirement is to have the addons to test installed in the database. One advantage of this approach is that even if your addon creates a new record for which another addon has created a new required field with a default value the tests will run correctly even if you don't depend on this addon.
  • ...
With all these guidelines in place, we improve the quality, reliability, maintainability, flexibility, compatibility and the speed of the tests in both development and integration levels. Last but not least, in our CI we are now able to run our tests in parallel by using the pytest-xdist plugin. This is a huge improvement in the speed of the tests.

We are still not perfect and since all the code modifying an existing behaviour is activated in production, some tests will fail if they are run on the production database since they are not designed to run with the modified behaviour.. But we solved a lot of issues we encountered in the past when tests were run by odoo at install/upgrade time. 

Regards,

lmi


Use post install test tags and remove at install. And refuse PRs with required fields that don't set defaults. This is the normal practice anyway as you would get errors with or without using demo data.

Also, one other benefit, particularly in number based tests is we use a library called hypothesis to determine values and try to find failing combos.

Le ven. 13 déc. 2024, 04:52, Raphaël Valyi <notifications@odoo-community.org> a écrit :
Hello Odoo community,

More and more the OCA contributors avoid writing tests which depend on demo data and so does Odoo SA. This makes it easier to ensure tests are simple and will run no matter how demo data might be touched by other modules.

But eventually this could bring other benefits such as the ability to run these standard tests in the CI with production database dumps (that is where the demo data cannot be loaded in general).

In this case, I have a first question: probably some customization's might add required fields. What would be the best practice to be able to run these "native" or OCA tests that might fail to create records with such new constraints? We could imagine some monkey patches and standard hooks to help... We kind of face this already today by running tests with all repo modules installed, but with production dumps, much more modules would be installed together along with some non OCA customization which also need to be developed more cheaply...

But more important is my second question: if we set it as a good practice that OCA tests shouldn't depend on demo data, why are we still running the OCA CI with demo data enabled (still the case with Odoo 18 CI) ? Or is it planned we stop doing this in Odoo 19? I mean we could also save a good deal of server resources/time by not loading the gazillions of catchy Odoo CE demo data in each test run...


Thank you very much for your answers.


--
Raphaël Valyi
Founder and consultant

_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe



--
Laurent Mignon
Technical lead / Management Team
T: +32 2 8883148


Atrium Building, Drève Richelle 167 | B-1410 Waterloo | Belgium
Val Benoit, Quai Banning 6 | B-4000 Liège | Belgium
Zone industrielle 22 | L-8287 Kehlen | Luxembourg

by Laurent Mignon - 09:06 - 13 Dec 2024

Reference

  • demo data and tests
    Hello Odoo community,

    More and more the OCA contributors avoid writing tests which depend on demo data and so does Odoo SA. This makes it easier to ensure tests are simple and will run no matter how demo data might be touched by other modules.

    But eventually this could bring other benefits such as the ability to run these standard tests in the CI with production database dumps (that is where the demo data cannot be loaded in general).

    In this case, I have a first question: probably some customization's might add required fields. What would be the best practice to be able to run these "native" or OCA tests that might fail to create records with such new constraints? We could imagine some monkey patches and standard hooks to help... We kind of face this already today by running tests with all repo modules installed, but with production dumps, much more modules would be installed together along with some non OCA customization which also need to be developed more cheaply...

    But more important is my second question: if we set it as a good practice that OCA tests shouldn't depend on demo data, why are we still running the OCA CI with demo data enabled (still the case with Odoo 18 CI) ? Or is it planned we stop doing this in Odoo 19? I mean we could also save a good deal of server resources/time by not loading the gazillions of catchy Odoo CE demo data in each test run...


    Thank you very much for your answers.


    --
    Raphaël Valyi
    Founder and consultant

    by "Raphaël Valyi" <rvalyi@akretion.com> - 04:51 - 12 Dec 2024

    4 replies 4 replies

    • Re: demo data and tests

      Hello,

      On my projects I try to use demo data for master data, for the following reasons:

      • It results in code that is more readable.
      • I disagree that creating object in the fixture makes it easier to ensure tests are simple. It's much simpler to include a «self.data_name = self.env.ref("")» statement in the fixture that to include a full create({}) call, which often requires the programmer to specify fields that have nothing to do with the features being tested (like the object's name, etc).
      • Simple code make it easier to avoid creating a common setUpClass() method for every testcase in a module, which WILL have unintended consequences when you need to test some novel codepath that depends on some flags.
      • Small setup differences that a testcase may need can be encoded in instructions following the «self.data_name = self.env.ref("")» statement, like this:
        • self.bank_journal_01 = self.env.ref("account_bugfixes.demo_bank_journal_01")
        • self.bank_journal_01.code = "MYCODE01"
      • Makes it easier to compare fixtures used in unrelated tests, seeing in what they defer, etc.
      • I disagree that not depending on demo data makes it easier to ensure tests will run no matter how demo data might be touched by other modules. Invariants about fixture, in my opinion, must be enforced by including additional tests that assert that the test data begins as expected. Also, it's a design principle that new modules must not break existing functionality, and so must happen for demo data (i.e. in general, one should avoid patching existing demo records in dependencies).
      • Tests that depend on demo data also run faster, because demo data is created only once.

      This is my opinion.

      João Jerónimo


      Às 15:52 de 12/12/2024, Raphaël Valyi escreveu:
      Hello Odoo community,

      More and more the OCA contributors avoid writing tests which depend on demo data and so does Odoo SA. This makes it easier to ensure tests are simple and will run no matter how demo data might be touched by other modules.

      But eventually this could bring other benefits such as the ability to run these standard tests in the CI with production database dumps (that is where the demo data cannot be loaded in general).

      In this case, I have a first question: probably some customization's might add required fields. What would be the best practice to be able to run these "native" or OCA tests that might fail to create records with such new constraints? We could imagine some monkey patches and standard hooks to help... We kind of face this already today by running tests with all repo modules installed, but with production dumps, much more modules would be installed together along with some non OCA customization which also need to be developed more cheaply...

      But more important is my second question: if we set it as a good practice that OCA tests shouldn't depend on demo data, why are we still running the OCA CI with demo data enabled (still the case with Odoo 18 CI) ? Or is it planned we stop doing this in Odoo 19? I mean we could also save a good deal of server resources/time by not loading the gazillions of catchy Odoo CE demo data in each test run...


      Thank you very much for your answers.


      --
      Raphaël Valyi
      Founder and consultant

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      by j_j_b_o_devel - 03:16 - 18 Dec 2024
    • Re: demo data and tests
      Hi

      On my projects tests are run at 2 levels:
      • At development level: The developer tests the code he has written to ensure that it works as expected. For this, the developper writes unit tests that are run on an empty database with only this addon installed. This ensures that the code is correct and works properly in the context defined by the dependencies of the addon.
      • At integration level: The CI runs all the tests on a database with all the addons installed. This ensures that the code is correct and works properly in the context of the whole system.

      To achieve this, we apply the following guidelines:
      • Tests don't rely on demo data; This eases the maintenance of the tests and ensures that the tests are not impacted by changes in the demo data. Moreover, it helps the developer to understand the context in which the tests are run.
      • Code that modifies / constrains an existing behavior must be disabled by default and must be enabled by a configuration parameter. This ensures that if you run the tests from other addons on a database where this addon is installed, the tests will not be impacted by this addon. It also ensures that if the tests are run at insall and another addon modifies the same behavior, the tests will not be impacted by this addon if it has been installed previously in the installation process. (The disabled behaviour is activated at test setup). All the disabled features are enabled in a top level addon which is excluded from the installation by the CI and installed only into the client database.
      •  Some specific test addons are developed for specific integration tests. These addons are useful to ensure that addons installed side by side are compatible even if they don't depend on each other. At test setup, we first activate the features of the addons that are activable by configuration parameter.
      • To avoid the need to use the "post_install' annotation, the CI runs the tests with pytest-odoo. The main difference compared to the way odoo runs the tests is that the registry is populated with all the addons installed into the database. pytest-odoo is a test runner only while when you run the tests with odoo, the tests are run along the installation/upgrade of the addons at the same time the registry is populated. To run tests with pytest-odoo, the first requirement is to have the addons to test installed in the database. One advantage of this approach is that even if your addon creates a new record for which another addon has created a new required field with a default value the tests will run correctly even if you don't depend on this addon.
      • ...
      With all these guidelines in place, we improve the quality, reliability, maintainability, flexibility, compatibility and the speed of the tests in both development and integration levels. Last but not least, in our CI we are now able to run our tests in parallel by using the pytest-xdist plugin. This is a huge improvement in the speed of the tests.

      We are still not perfect and since all the code modifying an existing behaviour is activated in production, some tests will fail if they are run on the production database since they are not designed to run with the modified behaviour.. But we solved a lot of issues we encountered in the past when tests were run by odoo at install/upgrade time. 

      Regards,

      lmi


      Use post install test tags and remove at install. And refuse PRs with required fields that don't set defaults. This is the normal practice anyway as you would get errors with or without using demo data.

      Also, one other benefit, particularly in number based tests is we use a library called hypothesis to determine values and try to find failing combos.

      Le ven. 13 déc. 2024, 04:52, Raphaël Valyi <notifications@odoo-community.org> a écrit :
      Hello Odoo community,

      More and more the OCA contributors avoid writing tests which depend on demo data and so does Odoo SA. This makes it easier to ensure tests are simple and will run no matter how demo data might be touched by other modules.

      But eventually this could bring other benefits such as the ability to run these standard tests in the CI with production database dumps (that is where the demo data cannot be loaded in general).

      In this case, I have a first question: probably some customization's might add required fields. What would be the best practice to be able to run these "native" or OCA tests that might fail to create records with such new constraints? We could imagine some monkey patches and standard hooks to help... We kind of face this already today by running tests with all repo modules installed, but with production dumps, much more modules would be installed together along with some non OCA customization which also need to be developed more cheaply...

      But more important is my second question: if we set it as a good practice that OCA tests shouldn't depend on demo data, why are we still running the OCA CI with demo data enabled (still the case with Odoo 18 CI) ? Or is it planned we stop doing this in Odoo 19? I mean we could also save a good deal of server resources/time by not loading the gazillions of catchy Odoo CE demo data in each test run...


      Thank you very much for your answers.


      --
      Raphaël Valyi
      Founder and consultant

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --
      Laurent Mignon
      Technical lead / Management Team
      T: +32 2 8883148


      Atrium Building, Drève Richelle 167 | B-1410 Waterloo | Belgium
      Val Benoit, Quai Banning 6 | B-4000 Liège | Belgium
      Zone industrielle 22 | L-8287 Kehlen | Luxembourg

      by Laurent Mignon - 09:06 - 13 Dec 2024
    • Re: demo data and tests
      Irt to loading demo data, afaik, post test it is a demo instance as well for manual testing

      Le ven. 13 déc. 2024, 06:28, Graeme Gellatly <graeme@moahub.nz> a écrit :
      Hi

      Use post install test tags and remove at install. And refuse PRs with required fields that don't set defaults. This is the normal practice anyway as you would get errors with or without using demo data.

      Also, one other benefit, particularly in number based tests is we use a library called hypothesis to determine values and try to find failing combos.

      Le ven. 13 déc. 2024, 04:52, Raphaël Valyi <notifications@odoo-community.org> a écrit :
      Hello Odoo community,

      More and more the OCA contributors avoid writing tests which depend on demo data and so does Odoo SA. This makes it easier to ensure tests are simple and will run no matter how demo data might be touched by other modules.

      But eventually this could bring other benefits such as the ability to run these standard tests in the CI with production database dumps (that is where the demo data cannot be loaded in general).

      In this case, I have a first question: probably some customization's might add required fields. What would be the best practice to be able to run these "native" or OCA tests that might fail to create records with such new constraints? We could imagine some monkey patches and standard hooks to help... We kind of face this already today by running tests with all repo modules installed, but with production dumps, much more modules would be installed together along with some non OCA customization which also need to be developed more cheaply...

      But more important is my second question: if we set it as a good practice that OCA tests shouldn't depend on demo data, why are we still running the OCA CI with demo data enabled (still the case with Odoo 18 CI) ? Or is it planned we stop doing this in Odoo 19? I mean we could also save a good deal of server resources/time by not loading the gazillions of catchy Odoo CE demo data in each test run...


      Thank you very much for your answers.


      --
      Raphaël Valyi
      Founder and consultant

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      by Graeme Gellatly - 06:31 - 12 Dec 2024
    • Re: demo data and tests
      Hi

      Use post install test tags and remove at install. And refuse PRs with required fields that don't set defaults. This is the normal practice anyway as you would get errors with or without using demo data.

      Also, one other benefit, particularly in number based tests is we use a library called hypothesis to determine values and try to find failing combos.

      Le ven. 13 déc. 2024, 04:52, Raphaël Valyi <notifications@odoo-community.org> a écrit :
      Hello Odoo community,

      More and more the OCA contributors avoid writing tests which depend on demo data and so does Odoo SA. This makes it easier to ensure tests are simple and will run no matter how demo data might be touched by other modules.

      But eventually this could bring other benefits such as the ability to run these standard tests in the CI with production database dumps (that is where the demo data cannot be loaded in general).

      In this case, I have a first question: probably some customization's might add required fields. What would be the best practice to be able to run these "native" or OCA tests that might fail to create records with such new constraints? We could imagine some monkey patches and standard hooks to help... We kind of face this already today by running tests with all repo modules installed, but with production dumps, much more modules would be installed together along with some non OCA customization which also need to be developed more cheaply...

      But more important is my second question: if we set it as a good practice that OCA tests shouldn't depend on demo data, why are we still running the OCA CI with demo data enabled (still the case with Odoo 18 CI) ? Or is it planned we stop doing this in Odoo 19? I mean we could also save a good deal of server resources/time by not loading the gazillions of catchy Odoo CE demo data in each test run...


      Thank you very much for your answers.


      --
      Raphaël Valyi
      Founder and consultant

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      by Graeme Gellatly - 06:31 - 12 Dec 2024