Skip to Content
  2. Mailing Lists
  3. Contributors
  4. Re: Full text search

Archives

Contributors

Re: Full text search

Re: Full text search

Re: Full text search

Hi Holger, 

Personally, I found your approach of using pure PostGreSQL insteresting here, given that elasticsearch requires an addtl service stack (docker container, or alike), and (as I learned the other day from Fred) Elasticsearch is developed in Java  (boo :-() Maybe not so much boo for me, as some of you know). For us, if connected with ITISeasy.docs, Solr comes as an option, too (built on lucene as well). 

Also a quick search retrieved this: https://rocky.dev/full-text-search. Performance is important. 

Have you by any chance - allready started covering some of the topics you marked as open in your presentation, like ranking, highlighting, proximity, parser tuning and dictionary training? 
Best, Joe 

Von: "Holger Brunn" <mail@hunki-enterprises.com>
An: "Odoo Community Association, (OCA) Contributors" <contributors@odoo-community.org>
Gesendet: Montag, 21. Juni 2021 18:46:45
Betreff: Re: Full text search

I did some work on this a long time ago in
https://github.com/Therp/fulltextsearch
That's basically a framework to make the postgres fulltext search operators 
usable via the Odoo UI. Something like this could still be helpful in v14 I 
think, but if I was tasked to start something from scratch I'd go the 
elasticsearch route nowadays I guess if there weren't any constraints.


-- 
Your partner for the hard Odoo problems
https://hunki-enterprises.com

_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe


by Joerg Lorenz. - 07:26 - 21 Jun 2021

Reference

  • Full text search
    Hello everyone,

    I remember some discussion about implementing full text search for Odoo, but I can't find any module or work in progress.

    Do you have any pointers?

    Thanks!

    --
    by Lorenzo Battistini. - 11:26 - 15 Jun 2021

    20 replies 20 replies

    • Re: Full text search
      Hi Lorenzo,
      I agree that having an implementation of postgres full text search would be a great improvement and would solve many use cases without needed integrations and other softwares.

      The Holger module could be a good starting point https://github.com/Therp/fulltextsearch

      I believe it would be great to make an implementation of full text search configurable by the user on each model similar as done here. Basically it would be:
      * user can choose on each models fields to be included on the full text search
      * user can choose to enable that search as the "name_search" one
      * user can choose to enable "smart search" it would be the first available option on the "search view" that would implement the full text search.

      So, for eg:
      * user choose on product.product the following fields to be used: name, barcode, default_code, brand
      * user enable "smart search" + " name search"
      * then:
      a) on a sale order, user can search a product with a very flexible way
      b) on the product.product tree view user can search with same flexible way

      All this searches are already implemented with native "ilike" searches but are not so powerful and performant. It's also integrated on ecommerce here

       


      Ing. Juan José Scarafía

      (+54 9 341) 3 278039

      twitter: @jjscarafia

      github: @jjscarafia



      El mar, 22 de jun. de 2021 a la(s) 06:22, Lorenzo Battistini (elbaddy@gmail.com) escribió:
      Thanks everyone for your helpful advice.

      My primary goal is to improve search mechanisms of some frontend pages, like e-commerce products or slides.
      Of course a solution at framework level, which would allow to apply the improved search to any model, would be great.

      I like the postgres approach because
      1. It seems powerful enough to cover our needs
      2. We don't have to install other (big) softwares, hard and onerous to maintain
      So, in the first place, I will have a look at Holger's and Anders's works.

      Thanks a lot!


      On Tue, 22 Jun 2021 at 08:12, Mignon, Laurent <laurent.mignon@acsone.eu> wrote:
      Hello everyone,

      I think Simone has given a good summary of the problem. I don't have much more to say. I would however just like to clarify a technical point about https://github.com/OCA/search-engine and more specifically the module dedicated to Elasticsearch. This module has been developed based of versions of Elasticsearch licensed under Apache 2.0. This module uses a limited number of methods from the api provided by Elasticsearch to publish and synchronize data in the indexes. I have personally used it with the Opendistro distribution of ES and I am confident that it will work seamlessly with Opensearh (https://opensearch.org/) which is a fork of ES based on the latest version released under Apache 2.0. If this is not the case, it is very likely that the adaptations to be made will be minor. These are just a few technical considerations of no importance until the functional needs are clearly defined.

      Regards,

      lmi


      On Tue, Jun 22, 2021 at 7:27 AM Simone Orsi <simahawk@gmail.com> wrote:
      Hello everyone,

      guys, let's get back on track! I think none of these topics is relevant for Lorenzo. At least not at this stage.
      Nobody is asking what's the final goal. Me included ;)

      @Lorenzo: What are you looking for?

      From my perspective:

      a) if you need something pluggable in the backend, something that works w/ "normal" odoo models you should have a look at what Holger did. It's a bit complex and many implementation details should be rewritten completely but it gives you an idea on how to get there.

      b) If you need something for the frontend, I'd definitely go for ES (especially if you want faceted search). Once you have ES in place you can use SearchKit JS to easily build a search UI.

      c) If you don't want to maintain the search engine and you want a reliable and easy to use configuration you can use Algolia (yes, a proprietary servivce! :P)  but watch out because it can be very expensive.

      Both b and c are feasible w/ the search-engine framework.

      d) If you only need a full text index for one very particular model in one very particular form, I'd go for basic postgres full text indexing + some custom code.

      HTH. If you need help on setting up the search engine stuff let me know ;)

      Bests,

      On Mon, Jun 21, 2021 at 8:57 PM Frederik Kramer <frederik.kramer@initos.com> wrote:
      Holger of course you are right, 

but just hiding crap in a container doesn't make it significantly less
cumbersome at the detailed level. But of course if you have a nicely
built docker-compose oder kubernetes service defintion at hand much is
done anywhere else an you only have to handle ENV variables.

Best
Frederik

Am Montag, den 21.06.2021, 18:31 +0000 schrieb Holger Brunn:





> > Neverthless all Java ;-)





> 





> when I was young I would complain about Java, but now you just stick





> it into a 





> container and be done with it





> _______________________________________________





> Mailing-List: https://odoo-community.org/groups/contributors-15





> Post to: mailto:contributors@odoo-community.org





> Unsubscribe: https://odoo-community.org/groups?unsubscribe





-- 
Dr.-Ing. Frederik Kramer
Geschäftsführer
        
initOS GmbH
An der Eisenbahn 1
21224 Rosengarten
        
Phone:  +49 4105 56156-12
Fax:    +49 4105 56156-10
Mobil:  +49 179 3901819
        
Email: frederik.kramer@initos.com
Web:   www.initos.com
        
Geschäftsführung:
Dr.-Ing. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke

Sitz der Gesellschaft: Rosengarten – Klecken
Amtsgericht Tostedt, HRB 205226
Steuer-Nr: 15/200/53247
USt-IdNr.: DE815580155

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --
      Simone Orsi

      Full stack Python web developer, Odoo specialist, Odoo Community Board Member, in love with open source.

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      by Juan José Scarafía - 04:16 - 23 Jun 2021
    • Re: Full text search
      Thanks everyone for your helpful advice.

      My primary goal is to improve search mechanisms of some frontend pages, like e-commerce products or slides.
      Of course a solution at framework level, which would allow to apply the improved search to any model, would be great.

      I like the postgres approach because
      1. It seems powerful enough to cover our needs
      2. We don't have to install other (big) softwares, hard and onerous to maintain
      So, in the first place, I will have a look at Holger's and Anders's works.

      Thanks a lot!


      On Tue, 22 Jun 2021 at 08:12, Mignon, Laurent <laurent.mignon@acsone.eu> wrote:
      Hello everyone,

      I think Simone has given a good summary of the problem. I don't have much more to say. I would however just like to clarify a technical point about https://github.com/OCA/search-engine and more specifically the module dedicated to Elasticsearch. This module has been developed based of versions of Elasticsearch licensed under Apache 2.0. This module uses a limited number of methods from the api provided by Elasticsearch to publish and synchronize data in the indexes. I have personally used it with the Opendistro distribution of ES and I am confident that it will work seamlessly with Opensearh (https://opensearch.org/) which is a fork of ES based on the latest version released under Apache 2.0. If this is not the case, it is very likely that the adaptations to be made will be minor. These are just a few technical considerations of no importance until the functional needs are clearly defined.

      Regards,

      lmi


      On Tue, Jun 22, 2021 at 7:27 AM Simone Orsi <simahawk@gmail.com> wrote:
      Hello everyone,

      guys, let's get back on track! I think none of these topics is relevant for Lorenzo. At least not at this stage.
      Nobody is asking what's the final goal. Me included ;)

      @Lorenzo: What are you looking for?

      From my perspective:

      a) if you need something pluggable in the backend, something that works w/ "normal" odoo models you should have a look at what Holger did. It's a bit complex and many implementation details should be rewritten completely but it gives you an idea on how to get there.

      b) If you need something for the frontend, I'd definitely go for ES (especially if you want faceted search). Once you have ES in place you can use SearchKit JS to easily build a search UI.

      c) If you don't want to maintain the search engine and you want a reliable and easy to use configuration you can use Algolia (yes, a proprietary servivce! :P)  but watch out because it can be very expensive.

      Both b and c are feasible w/ the search-engine framework.

      d) If you only need a full text index for one very particular model in one very particular form, I'd go for basic postgres full text indexing + some custom code.

      HTH. If you need help on setting up the search engine stuff let me know ;)

      Bests,

      On Mon, Jun 21, 2021 at 8:57 PM Frederik Kramer <frederik.kramer@initos.com> wrote:
      Holger of course you are right, 

but just hiding crap in a container doesn't make it significantly less
cumbersome at the detailed level. But of course if you have a nicely
built docker-compose oder kubernetes service defintion at hand much is
done anywhere else an you only have to handle ENV variables.

Best
Frederik

Am Montag, den 21.06.2021, 18:31 +0000 schrieb Holger Brunn:




> > Neverthless all Java ;-)




> 




> when I was young I would complain about Java, but now you just stick




> it into a 




> container and be done with it




> _______________________________________________




> Mailing-List: https://odoo-community.org/groups/contributors-15




> Post to: mailto:contributors@odoo-community.org




> Unsubscribe: https://odoo-community.org/groups?unsubscribe




-- 
Dr.-Ing. Frederik Kramer
Geschäftsführer
        
initOS GmbH
An der Eisenbahn 1
21224 Rosengarten
        
Phone:  +49 4105 56156-12
Fax:    +49 4105 56156-10
Mobil:  +49 179 3901819
        
Email: frederik.kramer@initos.com
Web:   www.initos.com
        
Geschäftsführung:
Dr.-Ing. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke

Sitz der Gesellschaft: Rosengarten – Klecken
Amtsgericht Tostedt, HRB 205226
Steuer-Nr: 15/200/53247
USt-IdNr.: DE815580155

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --
      Simone Orsi

      Full stack Python web developer, Odoo specialist, Odoo Community Board Member, in love with open source.

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --
      by Lorenzo Battistini. - 11:21 - 22 Jun 2021
    • Re: Full text search
      Hello everyone,

      I think Simone has given a good summary of the problem. I don't have much more to say. I would however just like to clarify a technical point about https://github.com/OCA/search-engine and more specifically the module dedicated to Elasticsearch. This module has been developed based of versions of Elasticsearch licensed under Apache 2.0. This module uses a limited number of methods from the api provided by Elasticsearch to publish and synchronize data in the indexes. I have personally used it with the Opendistro distribution of ES and I am confident that it will work seamlessly with Opensearh (https://opensearch.org/) which is a fork of ES based on the latest version released under Apache 2.0. If this is not the case, it is very likely that the adaptations to be made will be minor. These are just a few technical considerations of no importance until the functional needs are clearly defined.

      Regards,

      lmi


      On Tue, Jun 22, 2021 at 7:27 AM Simone Orsi <simahawk@gmail.com> wrote:
      Hello everyone,

      guys, let's get back on track! I think none of these topics is relevant for Lorenzo. At least not at this stage.
      Nobody is asking what's the final goal. Me included ;)

      @Lorenzo: What are you looking for?

      From my perspective:

      a) if you need something pluggable in the backend, something that works w/ "normal" odoo models you should have a look at what Holger did. It's a bit complex and many implementation details should be rewritten completely but it gives you an idea on how to get there.

      b) If you need something for the frontend, I'd definitely go for ES (especially if you want faceted search). Once you have ES in place you can use SearchKit JS to easily build a search UI.

      c) If you don't want to maintain the search engine and you want a reliable and easy to use configuration you can use Algolia (yes, a proprietary servivce! :P)  but watch out because it can be very expensive.

      Both b and c are feasible w/ the search-engine framework.

      d) If you only need a full text index for one very particular model in one very particular form, I'd go for basic postgres full text indexing + some custom code.

      HTH. If you need help on setting up the search engine stuff let me know ;)

      Bests,

      On Mon, Jun 21, 2021 at 8:57 PM Frederik Kramer <frederik.kramer@initos.com> wrote:
      Holger of course you are right, 

but just hiding crap in a container doesn't make it significantly less
cumbersome at the detailed level. But of course if you have a nicely
built docker-compose oder kubernetes service defintion at hand much is
done anywhere else an you only have to handle ENV variables.

Best
Frederik

Am Montag, den 21.06.2021, 18:31 +0000 schrieb Holger Brunn:



> > Neverthless all Java ;-)



> 



> when I was young I would complain about Java, but now you just stick



> it into a 



> container and be done with it



> _______________________________________________



> Mailing-List: https://odoo-community.org/groups/contributors-15



> Post to: mailto:contributors@odoo-community.org



> Unsubscribe: https://odoo-community.org/groups?unsubscribe



-- 
Dr.-Ing. Frederik Kramer
Geschäftsführer
        
initOS GmbH
An der Eisenbahn 1
21224 Rosengarten
        
Phone:  +49 4105 56156-12
Fax:    +49 4105 56156-10
Mobil:  +49 179 3901819
        
Email: frederik.kramer@initos.com
Web:   www.initos.com
        
Geschäftsführung:
Dr.-Ing. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke

Sitz der Gesellschaft: Rosengarten – Klecken
Amtsgericht Tostedt, HRB 205226
Steuer-Nr: 15/200/53247
USt-IdNr.: DE815580155

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --
      Simone Orsi

      Full stack Python web developer, Odoo specialist, Odoo Community Board Member, in love with open source.

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe

      by Laurent Mignon - 08:11 - 22 Jun 2021
    • Re: Full text search
      Hello everyone,

      guys, let's get back on track! I think none of these topics is relevant for Lorenzo. At least not at this stage.
      Nobody is asking what's the final goal. Me included ;)

      @Lorenzo: What are you looking for?

      From my perspective:

      a) if you need something pluggable in the backend, something that works w/ "normal" odoo models you should have a look at what Holger did. It's a bit complex and many implementation details should be rewritten completely but it gives you an idea on how to get there.

      b) If you need something for the frontend, I'd definitely go for ES (especially if you want faceted search). Once you have ES in place you can use SearchKit JS to easily build a search UI.

      c) If you don't want to maintain the search engine and you want a reliable and easy to use configuration you can use Algolia (yes, a proprietary servivce! :P)  but watch out because it can be very expensive.

      Both b and c are feasible w/ the search-engine framework.

      d) If you only need a full text index for one very particular model in one very particular form, I'd go for basic postgres full text indexing + some custom code.

      HTH. If you need help on setting up the search engine stuff let me know ;)

      Bests,

      On Mon, Jun 21, 2021 at 8:57 PM Frederik Kramer <frederik.kramer@initos.com> wrote:
      Holger of course you are right, 

but just hiding crap in a container doesn't make it significantly less
cumbersome at the detailed level. But of course if you have a nicely
built docker-compose oder kubernetes service defintion at hand much is
done anywhere else an you only have to handle ENV variables.

Best
Frederik

Am Montag, den 21.06.2021, 18:31 +0000 schrieb Holger Brunn:


> > Neverthless all Java ;-)


> 


> when I was young I would complain about Java, but now you just stick


> it into a 


> container and be done with it


> _______________________________________________


> Mailing-List: https://odoo-community.org/groups/contributors-15


> Post to: mailto:contributors@odoo-community.org


> Unsubscribe: https://odoo-community.org/groups?unsubscribe


-- 
Dr.-Ing. Frederik Kramer
Geschäftsführer
        
initOS GmbH
An der Eisenbahn 1
21224 Rosengarten
        
Phone:  +49 4105 56156-12
Fax:    +49 4105 56156-10
Mobil:  +49 179 3901819
        
Email: frederik.kramer@initos.com
Web:   www.initos.com
        
Geschäftsführung:
Dr.-Ing. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke

Sitz der Gesellschaft: Rosengarten – Klecken
Amtsgericht Tostedt, HRB 205226
Steuer-Nr: 15/200/53247
USt-IdNr.: DE815580155

      _______________________________________________
      Mailing-List: https://odoo-community.org/groups/contributors-15
      Post to: mailto:contributors@odoo-community.org
      Unsubscribe: https://odoo-community.org/groups?unsubscribe



      --
      Simone Orsi

      Full stack Python web developer, Odoo specialist, Odoo Community Board Member, in love with open source.
      by Simone Orsi - 07:26 - 22 Jun 2021
    • Re: Full text search 
      Holger of course you are right, 

but just hiding crap in a container doesn't make it significantly less
cumbersome at the detailed level. But of course if you have a nicely
built docker-compose oder kubernetes service defintion at hand much is
done anywhere else an you only have to handle ENV variables.

Best
Frederik

Am Montag, den 21.06.2021, 18:31 +0000 schrieb Holger Brunn:

> > Neverthless all Java ;-)

> 

> when I was young I would complain about Java, but now you just stick

> it into a 

> container and be done with it

> _______________________________________________

> Mailing-List: https://odoo-community.org/groups/contributors-15

> Post to: mailto:contributors@odoo-community.org

> Unsubscribe: https://odoo-community.org/groups?unsubscribe

-- 
Dr.-Ing. Frederik Kramer
Geschäftsführer
        
initOS GmbH
An der Eisenbahn 1
21224 Rosengarten
        
Phone:  +49 4105 56156-12
Fax:    +49 4105 56156-10
Mobil:  +49 179 3901819
        
Email: frederik.kramer@initos.com
Web:   www.initos.com
        
Geschäftsführung:
Dr.-Ing. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke

Sitz der Gesellschaft: Rosengarten – Klecken
Amtsgericht Tostedt, HRB 205226
Steuer-Nr: 15/200/53247
USt-IdNr.: DE815580155
      by Frederik Kramer - 08:55 - 21 Jun 2021