Skip to Content
  2. Mailing Lists
  3. Contributors
  4. Re: Large Data Files

Archives

Contributors

Re: Large Data Files

Re: Large Data Files

Re: Large Data Files

There's create_multi also in some Odoo versions, but which one are you on?

On 8/20/24 17:32, Jerôme Dewandre wrote:
Hello,

I am currently working on a syncro with a legacy system (adesoft) containing a large amount of data that must be synchronized on a daily basis (such as meetings).

It seems everything starts getting slow when I import 30.000 records with the conventional "create()" method.

I suppose the ORM might be an issue here. Potential workaround:

1. Bypass the ORM to create a record with self.env.cr.execute (but if I want to delete them I will also need a custom query)
2. Bypass the ORM with stored procedures (https://www.postgresql.org/docs/current/sql-createprocedure.html)
3. Increase the CPU/RAM/Worker nodes
4. Some better ideas?

What would be the best way to go?

A piece of my current test (df is a pandas dataframe containing the new events): 

@api.model
    def create_events_from_df(self, df):
        Event = self.env['event.event']
        events_data = []
        for _, row in df.iterrows():
            event_data = {
                'location': row['location'],
                'name': row['name'],
                'date_begin': row['date_begin'],
                'date_end': row['date_end'],
            }
            events_data.append(event_data)
       
        # Create all events in a single batch
        Event.create(events_data)

Thanks in advance if you read this, and thanks again if you replied :)

Jérôme

_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe

by Tom Blauwendraat - 05:51 - 20 Aug 2024

Reference

  • Large Data Files
    Hello,

    I am currently working on a syncro with a legacy system (adesoft) containing a large amount of data that must be synchronized on a daily basis (such as meetings).

    It seems everything starts getting slow when I import 30.000 records with the conventional "create()" method.

    I suppose the ORM might be an issue here. Potential workaround:

    1. Bypass the ORM to create a record with self.env.cr.execute (but if I want to delete them I will also need a custom query)
    2. Bypass the ORM with stored procedures (https://www.postgresql.org/docs/current/sql-createprocedure.html)
    3. Increase the CPU/RAM/Worker nodes
    4. Some better ideas?

    What would be the best way to go?

    A piece of my current test (df is a pandas dataframe containing the new events): 

    @api.model
        def create_events_from_df(self, df):
            Event = self.env['event.event']
            events_data = []
            for _, row in df.iterrows():
                event_data = {
                    'location': row['location'],
                    'name': row['name'],
                    'date_begin': row['date_begin'],
                    'date_end': row['date_end'],
                }
                events_data.append(event_data)
           
            # Create all events in a single batch
            Event.create(events_data)

    Thanks in advance if you read this, and thanks again if you replied :)

    Jérôme
    by "Jerôme Dewandre" <jerome.dewandre.mail@gmail.com> - 05:31 - 20 Aug 2024