Archives
Contributors
initialization of qweb template caches on website
Hello,
A customer (odoo 12.0) here is facing a performance issue related to
qweb templates on the website. The first time a worker serves these, we
get terrible performance. I suspect we are not the only ones facing
this. Using an external service to GET /shop for instance on a regular
basis sort of alleviates this, but this is really unelegant.
Is there a killer module I missed which is able to preload the cache for
a number of views each time a worker is started? Or some other strategy?
Thanks for your feedback
--
Alexandre Fayolle
Chef de Projet
Tel : +33 4 58 48 20 30
Camptocamp France SAS
18 rue du Lac Saint André
73 370 Le Bourget-du-Lac
France
http://www.camptocamp.com
by Alexandre Fayolle - 02:21 - 8 Jun 2020
Follow-Ups
-
Re: initialization of qweb template caches on website
Hi AlexandreI just checked the svg files attached in your previous email.The cache process is consuming 1/4 of the total time:In your profiling you are using 4 workers and the cache process was computed for 2 workers (pid-126 and pid-130)The issue is worse if you are using 20 workers so this cache will be computed 20 times.And even worse if you have a method clearing cache so often since that it will clear the cache of all methods in all workers.The following methods are clearing cache in Odoo:It means that if you create/unlink/modify a new crm.reveal.rule so the website will be slow again even if the crm.reveal.rule is not affecting the website.The same case for:The worst cases are cron of minutes clearing cache:- Check the following e.g. of a fixed issue related https://github.com/odoo/odoo/issues/25029But you still can have a cron calling a method clearing caches.Because of all this I have created the following issue:Check the Olivier Dony answer.Regards!El mar., 9 jun. 2020 a las 2:17, Alexandre Fayolle (<alexandre.fayolle@camptocamp.com>) escribió:Thanks for jumping on this Moises :) Yes I am able to reproduce locally, and I did run some profiling before asking :) Attached are 2 profiling sessions loading the same page with py-spy (https://pypi.org/project/py-spy/), once with an uninitialized worker, and once with an initialized worker. What I see is that a significant amount of time is spent in the flatten method (odoo/http.py:1270) in the uninitialized worker run, and this is not present in the initialized worker run (meaning the result of flatten is cached, and we see this in the call stack). The database is not on the critical path here, as far as I can tell, it's a pure python issue, and caching cannot directly help, because the performance issue is directly caused by the cache initialization. So what I would like to do is preload the cache when the worker starts. With a "normal" system, this would be done by browsing the site after it starts, but with Odoo's recycling workers, it is harder to do from outside. And besides, I found out that the caches are being invalidated, and we had to disable the event reminder cron which is invalidating all caches every 30min. I'll check with the customer if he is ok with sharing a database dump, and also with our GDPR guys here at Camptocamp. Alexandre On 09/06/2020 06:32, Moises Lopez wrote: > Hi Alexandre, > >> ...issue related to qweb templates on the website. *The first time a > worker serves these*... > > I agree with you > - If it is only the first time so it confirms that there is a slow > cached process. > - Since the first time was slow (caching) and the second one it is not. > (If the second time is using the same worker). > > 1) Can you reproduce it even if you are not using a proxy with cache > (e.g. nginx)? > > If we analyze Odoo we can look for the cached methods: > - rgrep "\@.*cache(" . --include=*.py --exclude-dir=test* -A2 |grep " > def " > > The following method are cached in Odoo and enterprise projects: > - https://gist.github.com/moylop260/a3394674e570da8ca457ef4fe2eb4176 > > In our experience slowest are: > - ir_rule._compute_domain: > - computing a lot of records > - computing relational fields (field_o2m.field_id.name > ) > - Using a computed field > - ir_translation.__get_source: > - A lot of terms to translate in the qweb view > - A lot of fields(translate=True) renderized > > But if you like to find the culprit... Are you able to run profilers? > > Check the following cases detected before using profilers: > - https://github.com/odoo/odoo/pull/49690 > - https://github.com/odoo/odoo/pull/25196 > > > - Enable advanced logs in postgresql to get a pgbadger report to detect: > - most executed queries > - slowest executed queries > - missing index > - locked records > - Run py-spy to get a flame to detect: > - slowest python methods > > Running both at the same time you can cross information and look for the > culprit easier. > > > El lun., 8 jun. 2020 a las 8:22, Gustavo Valverde > (<gustavo@iterativo.do>) escribió: > > Hi Alexandre, > > This is an interesting subject, as it's kind of hard to find a sweet > spot. > > I've been dealing with Odoo performance improvements in different > scenarios for a few months now. I thought I had most of it figured > out until I started dealing with ecommerce and fighting with > Google's pagespeed. > > I found out multiple things have to be taken into consideration when > dealing with websites: > > * If you have a high amount of workers, restarting or redeploying > new changes make things hellish slow. > * Mixed setups (multiprocess for the backend + multithread for the > website) make things better; but it depends on the load and > amount of online visitors that you could have at the same time. > * Nginx caching and CDNs are your friends. Don't be afraid of > caching /web/content, but do not cache it for excessive times. > > Regards, > > On Mon, Jun 8, 2020 at 8:22 AM Alexandre Fayolle > <alexandre.fayolle@camptocamp.com > <mailto:alexandre.fayolle@camptocamp.com>> wrote: > > Hello, > > A customer (odoo 12.0) here is facing a performance issue related to > qweb templates on the website. The first time a worker serves these, we > get terrible performance. I suspect we are not the only ones facing > this. Using an external service to GET /shop for instance on a regular > basis sort of alleviates this, but this is really unelegant. > > Is there a killer module I missed which is able to preload the cache for > a number of views each time a worker is started? Or some other strategy? > > Thanks for your feedback > > -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 > Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le > Bourget-du-Lac France http://www.camptocamp.com > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > <mailto:contributors@odoo-community.org> > Unsubscribe: https://odoo-community.org/groups?unsubscribe > > > > -- > > Gustavo Valverde > > Chief Technology Officer > > M(829) 686 8270 D (829) 956 5191 ext. 202 > > E gustavo@iterativo.do > > > > > ITERATIVO SRL > > Roble Corporate Center, Piso 7 > > Rafael Augusto Sánchez 86, Piantini, Santo Domingo, R. Dominicana > > > T(829) 956 5191| Einfo@iterativo.do | www.iterativo.do > > > > Member firm of ITERATIVO LLC & Progressa Corporate Group > > > Este mensaje y sus anexos pueden contener información confidencial y > privilegiada con la intención de que sea utilizada por las personas > u organizaciones a quienes esta dirigida, por lo que su uso es > exclusivo para su destinatario. Si usted ha recibido este mensaje > por error, favor de eliminarlo e informar al remitente del mensaje a > través de un correo de respuesta. Si este es el caso, le notificamos > que queda estrictamente prohibida la distribución o reproducción de > este e-mail y/o sus anexos. iterativo no se hace responsable de las > opiniones vertidas en esta comunicación que no estén acordes con su > quehacer y fines, y que no se revistan de un carácter oficial. > > This message and its enclosures may contain confidential and > privileged information intended for the use of people and > organizations to which it is directed and its use is thus limited to > its addressee. If you have received this message by mistake, please > eliminate it and inform the sender through a reply message. Should > this be the case, you are advised that the distribution or > reproduction of this e-mail and/or any attachments contained herein > is strictly forbidden. iterativo is not liable for opinions > expressed in this message which may not coincide with its > responsibilities and purpose and which may not express official > matters. > > __ __ > > https://iterativo.do/web/image/39802/hoja-verde.png____ > > /No imprimas este correo si no es necesario. Cuidemos del ambiente!/____ > > /Do not print this mail if it's not necessary. Let's take care of > the environment!/ > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > <mailto:contributors@odoo-community.org> > Unsubscribe: https://odoo-community.org/groups?unsubscribe > > > > -- > Moisés López Calderón > Mobile: (+521) 477-752-22-30 > Twitter: @moylop260 > hangout: moylop260@vauxoo.com <mailto:moylop260@vauxoo.com> > http://www.vauxoo.com - Odoo Gold Partner > Twitter: @vauxoo > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > Unsubscribe: https://odoo-community.org/groups?unsubscribe > -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le Bourget-du-Lac France http://www.camptocamp.com
_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
--Moisés López CalderónMobile: (+521) 477-752-22-30Twitter: @moylop260Twitter: @vauxoo
by Moisés López Calderón - 05:35 - 9 Jun 2020 -
Re: initialization of qweb template caches on website
Thanks for jumping on this Moises :) Yes I am able to reproduce locally, and I did run some profiling before asking :) Attached are 2 profiling sessions loading the same page with py-spy (https://pypi.org/project/py-spy/), once with an uninitialized worker, and once with an initialized worker. What I see is that a significant amount of time is spent in the flatten method (odoo/http.py:1270) in the uninitialized worker run, and this is not present in the initialized worker run (meaning the result of flatten is cached, and we see this in the call stack). The database is not on the critical path here, as far as I can tell, it's a pure python issue, and caching cannot directly help, because the performance issue is directly caused by the cache initialization. So what I would like to do is preload the cache when the worker starts. With a "normal" system, this would be done by browsing the site after it starts, but with Odoo's recycling workers, it is harder to do from outside. And besides, I found out that the caches are being invalidated, and we had to disable the event reminder cron which is invalidating all caches every 30min. I'll check with the customer if he is ok with sharing a database dump, and also with our GDPR guys here at Camptocamp. Alexandre On 09/06/2020 06:32, Moises Lopez wrote: > Hi Alexandre, > >> ...issue related to qweb templates on the website. *The first time a > worker serves these*... > > I agree with you > - If it is only the first time so it confirms that there is a slow > cached process. > - Since the first time was slow (caching) and the second one it is not. > (If the second time is using the same worker). > > 1) Can you reproduce it even if you are not using a proxy with cache > (e.g. nginx)? > > If we analyze Odoo we can look for the cached methods: > - rgrep "\@.*cache(" . --include=*.py --exclude-dir=test* -A2 |grep " > def " > > The following method are cached in Odoo and enterprise projects: > - https://gist.github.com/moylop260/a3394674e570da8ca457ef4fe2eb4176 > > In our experience slowest are: > - ir_rule._compute_domain: > - computing a lot of records > - computing relational fields (field_o2m.field_id.name > ) > - Using a computed field > - ir_translation.__get_source: > - A lot of terms to translate in the qweb view > - A lot of fields(translate=True) renderized > > But if you like to find the culprit... Are you able to run profilers? > > Check the following cases detected before using profilers: > - https://github.com/odoo/odoo/pull/49690 > - https://github.com/odoo/odoo/pull/25196 > > > - Enable advanced logs in postgresql to get a pgbadger report to detect: > - most executed queries > - slowest executed queries > - missing index > - locked records > - Run py-spy to get a flame to detect: > - slowest python methods > > Running both at the same time you can cross information and look for the > culprit easier. > > > El lun., 8 jun. 2020 a las 8:22, Gustavo Valverde > (<gustavo@iterativo.do>) escribió: > > Hi Alexandre, > > This is an interesting subject, as it's kind of hard to find a sweet > spot. > > I've been dealing with Odoo performance improvements in different > scenarios for a few months now. I thought I had most of it figured > out until I started dealing with ecommerce and fighting with > Google's pagespeed. > > I found out multiple things have to be taken into consideration when > dealing with websites: > > * If you have a high amount of workers, restarting or redeploying > new changes make things hellish slow. > * Mixed setups (multiprocess for the backend + multithread for the > website) make things better; but it depends on the load and > amount of online visitors that you could have at the same time. > * Nginx caching and CDNs are your friends. Don't be afraid of > caching /web/content, but do not cache it for excessive times. > > Regards, > > On Mon, Jun 8, 2020 at 8:22 AM Alexandre Fayolle > <alexandre.fayolle@camptocamp.com > <mailto:alexandre.fayolle@camptocamp.com>> wrote: > > Hello, > > A customer (odoo 12.0) here is facing a performance issue related to > qweb templates on the website. The first time a worker serves these, we > get terrible performance. I suspect we are not the only ones facing > this. Using an external service to GET /shop for instance on a regular > basis sort of alleviates this, but this is really unelegant. > > Is there a killer module I missed which is able to preload the cache for > a number of views each time a worker is started? Or some other strategy? > > Thanks for your feedback > > -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 > Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le > Bourget-du-Lac France http://www.camptocamp.com > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > <mailto:contributors@odoo-community.org> > Unsubscribe: https://odoo-community.org/groups?unsubscribe > > > > -- > > Gustavo Valverde > > Chief Technology Officer > > M(829) 686 8270 D (829) 956 5191 ext. 202 > > E gustavo@iterativo.do > > > > > ITERATIVO SRL > > Roble Corporate Center, Piso 7 > > Rafael Augusto Sánchez 86, Piantini, Santo Domingo, R. Dominicana > > > T(829) 956 5191| Einfo@iterativo.do | www.iterativo.do > > > > Member firm of ITERATIVO LLC & Progressa Corporate Group > > > Este mensaje y sus anexos pueden contener información confidencial y > privilegiada con la intención de que sea utilizada por las personas > u organizaciones a quienes esta dirigida, por lo que su uso es > exclusivo para su destinatario. Si usted ha recibido este mensaje > por error, favor de eliminarlo e informar al remitente del mensaje a > través de un correo de respuesta. Si este es el caso, le notificamos > que queda estrictamente prohibida la distribución o reproducción de > este e-mail y/o sus anexos. iterativo no se hace responsable de las > opiniones vertidas en esta comunicación que no estén acordes con su > quehacer y fines, y que no se revistan de un carácter oficial. > > This message and its enclosures may contain confidential and > privileged information intended for the use of people and > organizations to which it is directed and its use is thus limited to > its addressee. If you have received this message by mistake, please > eliminate it and inform the sender through a reply message. Should > this be the case, you are advised that the distribution or > reproduction of this e-mail and/or any attachments contained herein > is strictly forbidden. iterativo is not liable for opinions > expressed in this message which may not coincide with its > responsibilities and purpose and which may not express official > matters. > > __ __ > > https://iterativo.do/web/image/39802/hoja-verde.png____ > > /No imprimas este correo si no es necesario. Cuidemos del ambiente!/____ > > /Do not print this mail if it's not necessary. Let's take care of > the environment!/ > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > <mailto:contributors@odoo-community.org> > Unsubscribe: https://odoo-community.org/groups?unsubscribe > > > > -- > Moisés López Calderón > Mobile: (+521) 477-752-22-30 > Twitter: @moylop260 > hangout: moylop260@vauxoo.com <mailto:moylop260@vauxoo.com> > http://www.vauxoo.com - Odoo Gold Partner > Twitter: @vauxoo > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > Unsubscribe: https://odoo-community.org/groups?unsubscribe > -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le Bourget-du-Lac France http://www.camptocamp.com
by Alexandre Fayolle - 09:16 - 9 Jun 2020 -
Re: initialization of qweb template caches on website
Hi Alexandre,> ...issue related to qweb templates on the website. The first time a worker serves these...I agree with you- If it is only the first time so it confirms that there is a slow cached process.- Since the first time was slow (caching) and the second one it is not. (If the second time is using the same worker).1) Can you reproduce it even if you are not using a proxy with cache (e.g. nginx)?If we analyze Odoo we can look for the cached methods:- rgrep "\@.*cache(" . --include=*.py --exclude-dir=test* -A2 |grep " def "The following method are cached in Odoo and enterprise projects:In our experience slowest are:- ir_rule._compute_domain:- computing a lot of records- computing relational fields (field_o2m.field_id.name)- Using a computed field- ir_translation.__get_source:- A lot of terms to translate in the qweb view- A lot of fields(translate=True) renderizedBut if you like to find the culprit... Are you able to run profilers?Check the following cases detected before using profilers:- Enable advanced logs in postgresql to get a pgbadger report to detect:- most executed queries- slowest executed queries- missing index- locked records- Run py-spy to get a flame to detect:- slowest python methodsRunning both at the same time you can cross information and look for the culprit easier.El lun., 8 jun. 2020 a las 8:22, Gustavo Valverde (<gustavo@iterativo.do>) escribió:Hi Alexandre,
This is an interesting subject, as it's kind of hard to find a sweet spot.I've been dealing with Odoo performance improvements in different scenarios for a few months now. I thought I had most of it figured out until I started dealing with ecommerce and fighting with Google's pagespeed.I found out multiple things have to be taken into consideration when dealing with websites:- If you have a high amount of workers, restarting or redeploying new changes make things hellish slow.
- Mixed setups (multiprocess for the backend + multithread for the website) make things better; but it depends on the load and amount of online visitors that you could have at the same time.
- Nginx caching and CDNs are your friends. Don't be afraid of caching /web/content, but do not cache it for excessive times.
Regards,On Mon, Jun 8, 2020 at 8:22 AM Alexandre Fayolle <alexandre.fayolle@camptocamp.com> wrote:Hello, A customer (odoo 12.0) here is facing a performance issue related to qweb templates on the website. The first time a worker serves these, we get terrible performance. I suspect we are not the only ones facing this. Using an external service to GET /shop for instance on a regular basis sort of alleviates this, but this is really unelegant. Is there a killer module I missed which is able to preload the cache for a number of views each time a worker is started? Or some other strategy? Thanks for your feedback -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le Bourget-du-Lac France http://www.camptocamp.com_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
--Gustavo Valverde
Chief Technology Officer
M (829) 686 8270 D (829) 956 5191 ext. 202
E gustavo@iterativo.do
ITERATIVO SRL
Roble Corporate Center, Piso 7
Rafael Augusto Sánchez 86, Piantini, Santo Domingo, R. Dominicana
T (829) 956 5191| E info@iterativo.do | www.iterativo.do
Member firm of ITERATIVO LLC & Progressa Corporate Group
Este mensaje y sus anexos pueden contener información confidencial y privilegiada con la intención de que sea utilizada por las personas u organizaciones a quienes esta dirigida, por lo que su uso es exclusivo para su destinatario. Si usted ha recibido este mensaje por error, favor de eliminarlo e informar al remitente del mensaje a través de un correo de respuesta. Si este es el caso, le notificamos que queda estrictamente prohibida la distribución o reproducción de este e-mail y/o sus anexos. iterativo no se hace responsable de las opiniones vertidas en esta comunicación que no estén acordes con su quehacer y fines, y que no se revistan de un carácter oficial.
This message and its enclosures may contain confidential and privileged information intended for the use of people and organizations to which it is directed and its use is thus limited to its addressee. If you have received this message by mistake, please eliminate it and inform the sender through a reply message. Should this be the case, you are advised that the distribution or reproduction of this e-mail and/or any attachments contained herein is strictly forbidden. iterativo is not liable for opinions expressed in this message which may not coincide with its responsibilities and purpose and which may not express official matters.
No imprimas este correo si no es necesario. Cuidemos del ambiente!
Do not print this mail if it's not necessary. Let's take care of the environment!
_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
--Moisés López CalderónMobile: (+521) 477-752-22-30Twitter: @moylop260Twitter: @vauxoo
by Moisés López Calderón - 06:31 - 9 Jun 2020 -
Re: initialization of qweb template caches on website
Hi Alexandre,
This is an interesting subject, as it's kind of hard to find a sweet spot.I've been dealing with Odoo performance improvements in different scenarios for a few months now. I thought I had most of it figured out until I started dealing with ecommerce and fighting with Google's pagespeed.I found out multiple things have to be taken into consideration when dealing with websites:- If you have a high amount of workers, restarting or redeploying new changes make things hellish slow.
- Mixed setups (multiprocess for the backend + multithread for the website) make things better; but it depends on the load and amount of online visitors that you could have at the same time.
- Nginx caching and CDNs are your friends. Don't be afraid of caching /web/content, but do not cache it for excessive times.
Regards,On Mon, Jun 8, 2020 at 8:22 AM Alexandre Fayolle <alexandre.fayolle@camptocamp.com> wrote:Hello, A customer (odoo 12.0) here is facing a performance issue related to qweb templates on the website. The first time a worker serves these, we get terrible performance. I suspect we are not the only ones facing this. Using an external service to GET /shop for instance on a regular basis sort of alleviates this, but this is really unelegant. Is there a killer module I missed which is able to preload the cache for a number of views each time a worker is started? Or some other strategy? Thanks for your feedback -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le Bourget-du-Lac France http://www.camptocamp.com_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
--Gustavo Valverde
Chief Technology Officer
M (829) 686 8270 D (829) 956 5191 ext. 202
E gustavo@iterativo.do
ITERATIVO SRL
Roble Corporate Center, Piso 7
Rafael Augusto Sánchez 86, Piantini, Santo Domingo, R. Dominicana
T (829) 956 5191| E info@iterativo.do | www.iterativo.do
Member firm of ITERATIVO LLC & Progressa Corporate Group
Este mensaje y sus anexos pueden contener información confidencial y privilegiada con la intención de que sea utilizada por las personas u organizaciones a quienes esta dirigida, por lo que su uso es exclusivo para su destinatario. Si usted ha recibido este mensaje por error, favor de eliminarlo e informar al remitente del mensaje a través de un correo de respuesta. Si este es el caso, le notificamos que queda estrictamente prohibida la distribución o reproducción de este e-mail y/o sus anexos. iterativo no se hace responsable de las opiniones vertidas en esta comunicación que no estén acordes con su quehacer y fines, y que no se revistan de un carácter oficial.
This message and its enclosures may contain confidential and privileged information intended for the use of people and organizations to which it is directed and its use is thus limited to its addressee. If you have received this message by mistake, please eliminate it and inform the sender through a reply message. Should this be the case, you are advised that the distribution or reproduction of this e-mail and/or any attachments contained herein is strictly forbidden. iterativo is not liable for opinions expressed in this message which may not coincide with its responsibilities and purpose and which may not express official matters.
No imprimas este correo si no es necesario. Cuidemos del ambiente!
Do not print this mail if it's not necessary. Let's take care of the environment!
by Gustavo Valverde <gustavo@iterativo.do> - 03:21 - 8 Jun 2020
