Archives
Contributors
Warning: Using "numpy" library you will have weird errors in
production even if you are not using it
Hello OCA Contributors,
numpy is a powerful math library
But there are weird issues that could be a nightmare in production to found them.
Imagine that you start your odoo-bin and during starting you just see the following message:
2020-10-29 15:13:15,053 194 INFO openerp_test odoo.modules.loading: updating modules list
Segmentation fault (core dumped)
Segmentation fault (core dumped)
You can run gdb to find more details but it shows:
[New Thread 0x7fb752885700 (LWP 1630)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb752885700 (LWP 1630)]
0x0000000000000000 in ?? ()
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb752885700 (LWP 1630)]
0x0000000000000000 in ?? ()
Really, WT* is that?
Where is the issue man?
After 3 days of my weekend without sleep and running line by line I found that the impostor was: numpy
I just uninstalled and now all is working well.
The weird part is that this is a corner case issue and it is not reproduced easily.
Virtualization environment with a particular kind of processor.
In our case it was reproduced only and only in production environment (no staging ones)
Another issue was that all processors were used 100%
We just uninstalled numpy and now all is working well.
The issue is only using this line: "import numpy"
Even if you are not using that odoo module it line is executed and all your environment will be broken.
More info about the issues:
- Vx/MQT#315 (Big debug history)
- numpy#17674 (more recent)
- numpy#13059 (guilty I didn't sleep for 3 days)
The following libraries are using numpy as dependency:
- pandas
- bokeh
- altair
- websocket
So the following OCA projects depends of numpy:
- OCA/web
In fact, numpy is deprecating financial operations, we can use the following library:
IMHO we should add to "numpy" in the blacklist of libraries in the OCA
And look for other alternatives.
For pandas, I know it could increase the performance to process too much records
but in Odoo we have PostgreSQL and maybe we could find similar results.
Maybe a pipeline warning for our CI could be helpful if this library is installed.
What do you think?
Moisés López Calderón
Mobile: (+52) 477-752-22-30
Twitter: @moylop260
Twitter: @vauxoo
by Moisés López Calderón - 06:35 - 30 Oct 2020
Follow-Ups
-
Re: Warning: Using "numpy" library you will have weird errors in
production even if you are not using it
Thanks Moises for the heads up!I faced a similar case.A (non-OCA) module was using openpyxl,which loads numpy when available,which loads libopenblaspOpenBLAS creates a number of threads equal to the number of core threads available: 56 in my case (production server),so it quickly reached limit_memory_hardand the process was killed (SIGSEGV)Forcing OPENBLAS_NUM_THREADS=1 fixed the issue.Regards,On Mon, Nov 2, 2020 at 3:12 PM Stéphane Bidoul <stephane.bidoul@acsone.eu> wrote:Hi Moises,It's indeed unfortunate that you have had such issues. It's also very surprising that websocket-client attempts to import numpy :)But I don't think blacklisting numpy is appropriate. It is very widely used outside OCA and can be useful at times in OCA too.-sbiOn Mon, Nov 2, 2020 at 9:01 AM Pedro M. Baeza (Tecnativa) <pedro.baeza@tecnativa.com> wrote:Thanks for the warning.I don't use any OCA module using numpy AFAIK, but there are some out there indeed. Not sure if blacklisting is the best thing to do, as changing such modules to adapt to other things may not be easy, but developers that used it can tell.Regards._______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
by Nils Hamerlinck - 08:06 - 30 Jan 2021 -
Re: Warning: Using "numpy" library you will have weird errors in
production even if you are not using it
Hi Moises,It's indeed unfortunate that you have had such issues. It's also very surprising that websocket-client attempts to import numpy :)But I don't think blacklisting numpy is appropriate. It is very widely used outside OCA and can be useful at times in OCA too.-sbiOn Mon, Nov 2, 2020 at 9:01 AM Pedro M. Baeza (Tecnativa) <pedro.baeza@tecnativa.com> wrote:Thanks for the warning.I don't use any OCA module using numpy AFAIK, but there are some out there indeed. Not sure if blacklisting is the best thing to do, as changing such modules to adapt to other things may not be easy, but developers that used it can tell.Regards._______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
by Stéphane Bidoul - 09:11 - 2 Nov 2020 -
Re: Warning: Using "numpy" library you will have weird errors in
production even if you are not using it
Thanks for the warning.I don't use any OCA module using numpy AFAIK, but there are some out there indeed. Not sure if blacklisting is the best thing to do, as changing such modules to adapt to other things may not be easy, but developers that used it can tell.Regards.
by Pedro M. Baeza - 09:00 - 2 Nov 2020