Skip to Content

Contributors

Reducing the size of a database

Dear contributors,

Some of our customers have been using their database for more than 6 years. They want to reduce the size of the database, by archiving/aggregating old data (accounting entries, sales lines, stock movements, etc.). It would also be easier to do exports, pivot tables and so on.

I see different solutions :

1. Manually aggregate old data, with a specific method  by type of tables (account move lines, stock picking, sale order lines). For instance, aggregate old accounting entries with annual miscellaneous operations. Of course, we should keep a backup of the old data somewhere.

2. Mass delete all the old and unused records, such as canceled POs, articles created by mistake. I don't see a way to do this quickly.

Maybe an alternative solution would be a module that sets a "limit date" that acts as an automatic filter when doing exports, pivots, etc.

Do you know any tool or procedure that would help with this ?

Best regards,
-- 
Victor Champonnois - Coop IT Easy
Tel : +32 475 81 01 12

by Victor - 03:21 - 23 Jan 2024

Follow-Ups

  • Re: Reducing the size of a database
    Mail message table is typically 50 per cent of database or more. Frankly 6 years is not a lot. In my country fiscal authorities require 7 years, some other legal requirements need 50 years.

    On Wed, 24 Jan 2024, 5:36 am Holger Brunn, <notifications@odoo-community.org> wrote:
    > In my experience a
    
    
    > huge amount of records can be deleted by deleting old chatter
    
    
    > records.
    
    similar experience here, and for that we have
    https://github.com/OCA/server-tools/tree/14.0/autovacuum_message_attachment
    and its migrations to higher versions in the PRs.
    
    You should run the third query from
    https://wiki.postgresql.org/wiki/Disk_Usage
    anyways to first get an impression where the space goes.
    
    
    
    -- 
    Your partner for the hard Odoo problems
    https://hunki-enterprises.com

    _______________________________________________
    Mailing-List: https://odoo-community.org/groups/contributors-15
    Post to: mailto:contributors@odoo-community.org
    Unsubscribe: https://odoo-community.org/groups?unsubscribe


    by "Graeme Gellatly" <graeme@moahub.nz> - 09:25 - 23 Jan 2024
  • Re: Reducing the size of a database
    > In my experience a
    
    > huge amount of records can be deleted by deleting old chatter
    
    > records.
    
    similar experience here, and for that we have
    https://github.com/OCA/server-tools/tree/14.0/autovacuum_message_attachment
    and its migrations to higher versions in the PRs.
    
    You should run the third query from
    https://wiki.postgresql.org/wiki/Disk_Usage
    anyways to first get an impression where the space goes.
    
    
    -- 
    Your partner for the hard Odoo problems
    https://hunki-enterprises.com

    by Holger Brunn - 05:35 - 23 Jan 2024
  • Re: Reducing the size of a database

    Hi,


    In my experience a huge amount of records can be deleted by deleting old chatter records.


    Kind regards, Ronald


    On 23-01-2024 15:57, Florian Laporte wrote:

    Dear Victor,

    How large are these databases? Database size usually does not become problematic as a result of too many sale orders or stock movements. You would need an insane amount.
    If this is the case, and you’ve verified that the size is not caused by large blobs/filestore, the only thing you can do is mass-delete records based on some arbitrary parameter like their create_date or write_date. There are many records Odoo won’t let you delete, though. It will ask you to archive them instead, because they are still referenced elsewhere. Archiving will obviously do nothing for database size.

    Sincerely,

    Florian L


    On Tue, 23 Jan 2024 at 15:22, Victor Champonnois <notifications@odoo-community.org> wrote:

    Dear contributors,

    Some of our customers have been using their database for more than 6 years. They want to reduce the size of the database, by archiving/aggregating old data (accounting entries, sales lines, stock movements, etc.). It would also be easier to do exports, pivot tables and so on.

    I see different solutions :

    1. Manually aggregate old data, with a specific method  by type of tables (account move lines, stock picking, sale order lines). For instance, aggregate old accounting entries with annual miscellaneous operations. Of course, we should keep a backup of the old data somewhere.

    2. Mass delete all the old and unused records, such as canceled POs, articles created by mistake. I don't see a way to do this quickly.

    Maybe an alternative solution would be a module that sets a "limit date" that acts as an automatic filter when doing exports, pivots, etc.

    Do you know any tool or procedure that would help with this ?

    Best regards,
    -- 
    Victor Champonnois - Coop IT Easy
    Tel : +32 475 81 01 12

    _______________________________________________
    Mailing-List: https://odoo-community.org/groups/contributors-15
    Post to: mailto:contributors@odoo-community.org
    Unsubscribe: https://odoo-community.org/groups?unsubscribe

    _______________________________________________
    Mailing-List: https://odoo-community.org/groups/contributors-15
    Post to: mailto:contributors@odoo-community.org
    Unsubscribe: https://odoo-community.org/groups?unsubscribe


    by "Ronald Portier" <rportier@therp.nl> - 04:21 - 23 Jan 2024
  • Re: Reducing the size of a database
    I've seen this module:
    
    https://apps.odoo.com/apps/modules/13.0/muk_autovacuum/
    
    Obviously this can't be used on just any models, but for some models I 
    guess such an approach can work.
    
    Another idea I've proposed to a customer once, was to have a moment at 
    end of year where you create an "archive copy instance" for that year, 
    where people can login but not change anything. The code at that point 
    is also frozen and copied with. It's a dirty approach, as you'll after N 
    years be sitting with N archive instances, but there's a chance that at 
    some point they'll start agreeing to delete the oldest instance (as at 
    least here in NL, there's a year limit to which you are obliged to keep 
    financial history). The upside to this approach is that you can 
    aggressively start deleting records from live.
    
    -Tom
    
    
    

    by Tom Blauwendraat - 04:20 - 23 Jan 2024
  • Re: Reducing the size of a database

    Dear Victor,

    How large are these databases? Database size usually does not become problematic as a result of too many sale orders or stock movements. You would need an insane amount.
    If this is the case, and you’ve verified that the size is not caused by large blobs/filestore, the only thing you can do is mass-delete records based on some arbitrary parameter like their create_date or write_date. There are many records Odoo won’t let you delete, though. It will ask you to archive them instead, because they are still referenced elsewhere. Archiving will obviously do nothing for database size.

    Sincerely,

    Florian L


    On Tue, 23 Jan 2024 at 15:22, Victor Champonnois <notifications@odoo-community.org> wrote:

    Dear contributors,

    Some of our customers have been using their database for more than 6 years. They want to reduce the size of the database, by archiving/aggregating old data (accounting entries, sales lines, stock movements, etc.). It would also be easier to do exports, pivot tables and so on.

    I see different solutions :

    1. Manually aggregate old data, with a specific method  by type of tables (account move lines, stock picking, sale order lines). For instance, aggregate old accounting entries with annual miscellaneous operations. Of course, we should keep a backup of the old data somewhere.

    2. Mass delete all the old and unused records, such as canceled POs, articles created by mistake. I don't see a way to do this quickly.

    Maybe an alternative solution would be a module that sets a "limit date" that acts as an automatic filter when doing exports, pivots, etc.

    Do you know any tool or procedure that would help with this ?

    Best regards,
    -- 
    Victor Champonnois - Coop IT Easy
    Tel : +32 475 81 01 12

    _______________________________________________
    Mailing-List: https://odoo-community.org/groups/contributors-15
    Post to: mailto:contributors@odoo-community.org
    Unsubscribe: https://odoo-community.org/groups?unsubscribe


    by florian.laporte - 03:56 - 23 Jan 2024