Skip to Content
  2. Mailing Lists
  3. Contributors
  4. Re: Warning: Using "numpy" library you will have weird errors in production even if you are not using it

Archives

Contributors

Re: Warning: Using "numpy" library you will have weird errors in production even if you are not using it

Re: Warning: Using "numpy" library you will have weird errors in production even if you are not using it

Thanks Moises for the heads up!

I faced a similar case.

A (non-OCA) module was using openpyxl,
which loads numpy when available,
which loads libopenblasp

OpenBLAS creates a number of threads equal to the number of core threads available: 56 in my case (production server),
so it quickly reached limit_memory_hard
and the process was killed (SIGSEGV)

Forcing OPENBLAS_NUM_THREADS=1 fixed the issue.

Regards,

On Mon, Nov 2, 2020 at 3:12 PM Stéphane Bidoul <stephane.bidoul@acsone.eu> wrote:
Hi Moises,

It's indeed unfortunate that you have had such issues. It's also very surprising that websocket-client attempts to import numpy :)

But I don't think blacklisting numpy is appropriate. It is very widely used outside OCA and can be useful at times in OCA too.

-sbi

On Mon, Nov 2, 2020 at 9:01 AM Pedro M. Baeza (Tecnativa) <pedro.baeza@tecnativa.com> wrote:
Thanks for the warning.

I don't use any OCA module using numpy AFAIK, but there are some out there indeed. Not sure if blacklisting is the best thing to do, as changing such modules to adapt to other things may not be easy, but developers that used it can tell.

Regards.

_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe

by Nils Hamerlinck - 08:06 - 30 Jan 2021

Reference

  • Warning: Using "numpy" library you will have weird errors in production even if you are not using it
    Hello OCA Contributors,

    numpy is a powerful math library
    But there are weird issues that could be a nightmare in production to found them.

    Imagine that you start your odoo-bin and during starting you just see the following message:

        2020-10-29 15:13:15,053 194 INFO openerp_test odoo.modules.loading: updating modules list
        Segmentation fault (core dumped)

    You can run gdb to find more details but it shows:

        [New Thread 0x7fb752885700 (LWP 1630)]
        Program received signal SIGSEGV, Segmentation fault.
        [Switching to Thread 0x7fb752885700 (LWP 1630)]
        0x0000000000000000 in ?? ()

    Really, WT* is that?
    Where is the issue man?

    After 3 days of my weekend without sleep and running line by line I found that the impostor was: numpy

    I just uninstalled and now all is working well.

    The weird part is that this is a corner case issue and it is not reproduced easily.
    Virtualization environment with a particular kind of processor.
    In our case it was reproduced only and only in production environment (no staging ones)

    Another issue was that all processors were used 100%
    We just uninstalled numpy and now all is working well.

    The issue is only using this line: "import numpy"
    Even if you are not using that odoo module it line is executed and all your environment will be broken.

    More info about the issues:
     - Vx/MQT#315 (Big debug history)
     - numpy#17674 (more recent)
     - numpy#13059 (guilty I didn't sleep for 3 days)

    The following libraries are using numpy as dependency:
     - pandas
     - bokeh
     - altair
     - websocket

    So the following OCA projects depends of numpy:
     - OCA/web


    In fact, numpy is deprecating financial operations, we can use the following library:

    IMHO we should add to "numpy" in the blacklist of libraries in the OCA
    And look for other alternatives.
    For pandas, I know it could increase the performance to process too much records
    but in Odoo we have PostgreSQL and maybe we could find similar results.

    Maybe a pipeline warning for our CI could be helpful if this library is installed.

    What do you think?

    --
    Moisés López Calderón
    Mobile: (+52) 477-752-22-30
    Twitter: @moylop260
    hangout: moylop260@vauxoo.com
    http://www.vauxoo.com - Odoo Gold Partner
    Twitter: @vauxoo
    by Moisés López Calderón - 06:35 - 30 Oct 2020