metabase: Add upgrade from v2 to v3 #1334

Merged
fyrchik merged 1 commit from dstepanov-yadro/frostfs-node:feat/metabase_upgrade into master 2024-09-04 19:51:11 +00:00

Add upgrade from v2 to v3 metabase.

Tested on metabase file generated with TestGenerateMetabaseFile on support/v0.42 branch.
Metabase file size before upgrade 2.7 GB, after - 1.5 GB (4 million objects). Elapsed time on my laptop - 30 sec.

time ./bin/frostfs-adm metabase upgrade --path "/home/dstepanov/metabase_copy.v2"
2024-08-28T21:51:40+03:00:filling expiration epoch buckets...
2024-08-28T21:51:40+03:00:expiration epoch filled for 50000 objects...
2024-08-28T21:51:41+03:00:expiration epoch filled for 100000 objects...
2024-08-28T21:51:41+03:00:expiration epoch filled for 150000 objects...
2024-08-28T21:51:41+03:00:expiration epoch filled for 200000 objects...
2024-08-28T21:51:42+03:00:expiration epoch filled for 250000 objects...
2024-08-28T21:51:42+03:00:expiration epoch filled for 300000 objects...
2024-08-28T21:51:43+03:00:expiration epoch filled for 350000 objects...
2024-08-28T21:51:43+03:00:expiration epoch filled for 400000 objects...
2024-08-28T21:51:44+03:00:expiration epoch filled for 450000 objects...
2024-08-28T21:51:44+03:00:expiration epoch filled for 500000 objects...
2024-08-28T21:51:45+03:00:expiration epoch filled for 550000 objects...
2024-08-28T21:51:46+03:00:expiration epoch filled for 600000 objects...
2024-08-28T21:51:46+03:00:expiration epoch filled for 650000 objects...
2024-08-28T21:51:47+03:00:expiration epoch filled for 700000 objects...
2024-08-28T21:51:47+03:00:expiration epoch filled for 750000 objects...
2024-08-28T21:51:48+03:00:expiration epoch filled for 800000 objects...
2024-08-28T21:51:48+03:00:expiration epoch filled for 850000 objects...
2024-08-28T21:51:49+03:00:expiration epoch filled for 900000 objects...
2024-08-28T21:51:49+03:00:expiration epoch filled for 950000 objects...
2024-08-28T21:51:50+03:00:expiration epoch filled for 1000000 objects...
2024-08-28T21:51:50+03:00:expiration epoch filled for 1050000 objects...
2024-08-28T21:51:51+03:00:expiration epoch filled for 1100000 objects...
2024-08-28T21:51:51+03:00:expiration epoch filled for 1150000 objects...
2024-08-28T21:51:52+03:00:expiration epoch filled for 1200000 objects...
2024-08-28T21:51:52+03:00:expiration epoch filled for 1250000 objects...
2024-08-28T21:51:53+03:00:expiration epoch filled for 1300000 objects...
2024-08-28T21:51:53+03:00:expiration epoch filled for 1350000 objects...
2024-08-28T21:51:54+03:00:expiration epoch filled for 1400000 objects...
2024-08-28T21:51:54+03:00:expiration epoch filled for 1450000 objects...
2024-08-28T21:51:55+03:00:expiration epoch filled for 1500000 objects...
2024-08-28T21:51:56+03:00:expiration epoch filled for 1550000 objects...
2024-08-28T21:51:56+03:00:expiration epoch filled for 1600000 objects...
2024-08-28T21:51:56+03:00:filling expiration epoch buckets completed successfully, total 1600000 objects
2024-08-28T21:51:56+03:00:user attributes: deleting buckets...
2024-08-28T21:51:56+03:00:user attributes: deleting buckets completed successfully, deleted 20000 buckets
2024-08-28T21:51:56+03:00:owner ID index: deleting buckets...
2024-08-28T21:51:58+03:00:owner ID index: deleting buckets completed successfully, deleted 10000 buckets
2024-08-28T21:51:58+03:00:payload checksum: deleting buckets...
2024-08-28T21:51:58+03:00:payload checksum: deleting buckets completed successfully, deleted 10000 buckets
2024-08-28T21:51:58+03:00:compacting metabase...
2024-08-28T21:52:09+03:00:metabase compacted

real    0m29,593s
user    0m31,761s
sys     0m7,283s

frostfs-adm output examples:

/home/service/frostfs-adm-dstepanov metabase upgrade -c /etc/frostfs/storage/config.yml --path /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db
found 1 metabases:
1 : /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : open metabase: timeout
/srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : failed
/home/service/frostfs-adm-dstepanov metabase upgrade -c /etc/frostfs/storage/config.yml --config-dir /etc/frostfs/storage/conf.d/
found 12 metabases:
1 : /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db
2 : /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db
3 : /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db
4 : /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db
5 : /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db
6 : /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db
7 : /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db
8 : /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db
9 : /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db
10 : /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db
11 : /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db
12 : /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db : open metabase: timeout
upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db ...
error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db : open metabase: timeout
/srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db : failed
/srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db : failed
/srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db : failed
/srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db : failed
/srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db : failed
/srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db : failed
/srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db : failed
/srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db : failed
/srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : failed
/srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db : failed
/srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db : failed
/srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db : failed

/home/service/frostfs-adm-dstepanov metabase upgrade -c /etc/frostfs/storage/config.yml --config-dir /etc/frostfs/storage/conf.d/
found 12 metabases:
1 : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db
2 : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db
3 : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db
4 : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db
5 : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db
6 : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db
7 : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db
8 : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db
9 : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db
10 : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db
11 : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db
12 : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db
upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db ...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : user attributes: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : owner ID index: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : payload checksum: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : compacting metabase...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : metabase compacted
metabase /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db ...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : compacting metabase...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : metabase compacted
metabase /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db ...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : compacting metabase...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : metabase compacted
metabase /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db ...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : compacting metabase...
2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : metabase compacted
metabase /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db ...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : compacting metabase...
2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : metabase compacted
metabase /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : user attributes: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : owner ID index: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : payload checksum: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : metabase compacted
metabase /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : user attributes: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : owner ID index: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : payload checksum: deleting buckets completed successfully, deleted 1 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : metabase compacted
metabase /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : metabase compacted
metabase /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : metabase compacted
metabase /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : metabase compacted
metabase /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : metabase compacted
metabase /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db upgraded successfully
upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db ...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : filling expiration epoch buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : user attributes: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : owner ID index: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : payload checksum: deleting buckets...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : compacting metabase...
2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : metabase compacted
metabase /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db upgraded successfully
/srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : success
/srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : success
/srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : success
/srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : success
/srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : success
/srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : success
/srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : success
/srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : success
/srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : success
/srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : success
/srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : success
/srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : success
Add upgrade from v2 to v3 metabase. Tested on metabase file generated with `TestGenerateMetabaseFile` on `support/v0.42` branch. Metabase file size before upgrade 2.7 GB, after - 1.5 GB (4 million objects). Elapsed time on my laptop - 30 sec. ``` time ./bin/frostfs-adm metabase upgrade --path "/home/dstepanov/metabase_copy.v2" 2024-08-28T21:51:40+03:00:filling expiration epoch buckets... 2024-08-28T21:51:40+03:00:expiration epoch filled for 50000 objects... 2024-08-28T21:51:41+03:00:expiration epoch filled for 100000 objects... 2024-08-28T21:51:41+03:00:expiration epoch filled for 150000 objects... 2024-08-28T21:51:41+03:00:expiration epoch filled for 200000 objects... 2024-08-28T21:51:42+03:00:expiration epoch filled for 250000 objects... 2024-08-28T21:51:42+03:00:expiration epoch filled for 300000 objects... 2024-08-28T21:51:43+03:00:expiration epoch filled for 350000 objects... 2024-08-28T21:51:43+03:00:expiration epoch filled for 400000 objects... 2024-08-28T21:51:44+03:00:expiration epoch filled for 450000 objects... 2024-08-28T21:51:44+03:00:expiration epoch filled for 500000 objects... 2024-08-28T21:51:45+03:00:expiration epoch filled for 550000 objects... 2024-08-28T21:51:46+03:00:expiration epoch filled for 600000 objects... 2024-08-28T21:51:46+03:00:expiration epoch filled for 650000 objects... 2024-08-28T21:51:47+03:00:expiration epoch filled for 700000 objects... 2024-08-28T21:51:47+03:00:expiration epoch filled for 750000 objects... 2024-08-28T21:51:48+03:00:expiration epoch filled for 800000 objects... 2024-08-28T21:51:48+03:00:expiration epoch filled for 850000 objects... 2024-08-28T21:51:49+03:00:expiration epoch filled for 900000 objects... 2024-08-28T21:51:49+03:00:expiration epoch filled for 950000 objects... 2024-08-28T21:51:50+03:00:expiration epoch filled for 1000000 objects... 2024-08-28T21:51:50+03:00:expiration epoch filled for 1050000 objects... 2024-08-28T21:51:51+03:00:expiration epoch filled for 1100000 objects... 2024-08-28T21:51:51+03:00:expiration epoch filled for 1150000 objects... 2024-08-28T21:51:52+03:00:expiration epoch filled for 1200000 objects... 2024-08-28T21:51:52+03:00:expiration epoch filled for 1250000 objects... 2024-08-28T21:51:53+03:00:expiration epoch filled for 1300000 objects... 2024-08-28T21:51:53+03:00:expiration epoch filled for 1350000 objects... 2024-08-28T21:51:54+03:00:expiration epoch filled for 1400000 objects... 2024-08-28T21:51:54+03:00:expiration epoch filled for 1450000 objects... 2024-08-28T21:51:55+03:00:expiration epoch filled for 1500000 objects... 2024-08-28T21:51:56+03:00:expiration epoch filled for 1550000 objects... 2024-08-28T21:51:56+03:00:expiration epoch filled for 1600000 objects... 2024-08-28T21:51:56+03:00:filling expiration epoch buckets completed successfully, total 1600000 objects 2024-08-28T21:51:56+03:00:user attributes: deleting buckets... 2024-08-28T21:51:56+03:00:user attributes: deleting buckets completed successfully, deleted 20000 buckets 2024-08-28T21:51:56+03:00:owner ID index: deleting buckets... 2024-08-28T21:51:58+03:00:owner ID index: deleting buckets completed successfully, deleted 10000 buckets 2024-08-28T21:51:58+03:00:payload checksum: deleting buckets... 2024-08-28T21:51:58+03:00:payload checksum: deleting buckets completed successfully, deleted 10000 buckets 2024-08-28T21:51:58+03:00:compacting metabase... 2024-08-28T21:52:09+03:00:metabase compacted real 0m29,593s user 0m31,761s sys 0m7,283s ``` frostfs-adm output examples: ``` /home/service/frostfs-adm-dstepanov metabase upgrade -c /etc/frostfs/storage/config.yml --path /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db found 1 metabases: 1 : /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : open metabase: timeout /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : failed ``` ``` /home/service/frostfs-adm-dstepanov metabase upgrade -c /etc/frostfs/storage/config.yml --config-dir /etc/frostfs/storage/conf.d/ found 12 metabases: 1 : /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db 2 : /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db 3 : /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db 4 : /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db 5 : /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db 6 : /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db 7 : /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db 8 : /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db 9 : /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db 10 : /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db 11 : /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db 12 : /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db : open metabase: timeout upgrading metabase /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db ... error: failed to upgrade metabase /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db : open metabase: timeout /srv/frostfs/meta/fe4226ed6854478c/meta/08436168c92c115b/metabase.db : failed /srv/frostfs/meta/359ec68533b576d7/meta/733d76dc69550835/metabase.db : failed /srv/frostfs/meta/fe4226ed6854478c/meta/742016eb6c65474e/metabase.db : failed /srv/frostfs/meta/fe4226ed6854478c/meta/c79df0d2d87ad331/metabase.db : failed /srv/frostfs/meta/fe4226ed6854478c/meta/66e5b4433280b084/metabase.db : failed /srv/frostfs/meta/359ec68533b576d7/meta/7557f81582ba284b/metabase.db : failed /srv/frostfs/meta/fe4226ed6854478c/meta/9260748033f302b7/metabase.db : failed /srv/frostfs/meta/359ec68533b576d7/meta/c39c6014e598533a/metabase.db : failed /srv/frostfs/meta/359ec68533b576d7/meta/033801f339271580/metabase.db : failed /srv/frostfs/meta/359ec68533b576d7/meta/129a2506078f0148/metabase.db : failed /srv/frostfs/meta/fe4226ed6854478c/meta/28717d88c70f0c31/metabase.db : failed /srv/frostfs/meta/359ec68533b576d7/meta/4ee39c1ea74d4b42/metabase.db : failed ``` ``` /home/service/frostfs-adm-dstepanov metabase upgrade -c /etc/frostfs/storage/config.yml --config-dir /etc/frostfs/storage/conf.d/ found 12 metabases: 1 : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db 2 : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db 3 : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db 4 : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db 5 : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db 6 : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db 7 : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db 8 : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db 9 : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db 10 : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db 11 : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db 12 : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db ... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : user attributes: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : owner ID index: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : payload checksum: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : compacting metabase... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : metabase compacted metabase /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db ... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : compacting metabase... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : metabase compacted metabase /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db ... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : compacting metabase... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : metabase compacted metabase /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db ... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : compacting metabase... 2024-08-30T06:10:56Z : /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : metabase compacted metabase /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db ... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : compacting metabase... 2024-08-30T06:10:56Z : /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : metabase compacted metabase /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : user attributes: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : owner ID index: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : payload checksum: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : metabase compacted metabase /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : user attributes: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : owner ID index: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : payload checksum: deleting buckets completed successfully, deleted 1 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : metabase compacted metabase /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : metabase compacted metabase /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : metabase compacted metabase /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : metabase compacted metabase /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : metabase compacted metabase /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db upgraded successfully upgrading metabase /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db ... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : filling expiration epoch buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : filling expiration epoch buckets completed successfully, total 0 objects 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : user attributes: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : user attributes: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : owner ID index: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : owner ID index: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : payload checksum: deleting buckets... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : payload checksum: deleting buckets completed successfully, deleted 0 buckets 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : compacting metabase... 2024-08-30T06:10:57Z : /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : metabase compacted metabase /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db upgraded successfully /srv/frostfs/meta/4721dcb842043023/meta/5a9885360764de2f/metabase.db : success /srv/frostfs/meta/83e4890651d57696/meta/6141bc748220ba0e/metabase.db : success /srv/frostfs/meta/4721dcb842043023/meta/67999fdf28b37c51/metabase.db : success /srv/frostfs/meta/83e4890651d57696/meta/703b548398509fbc/metabase.db : success /srv/frostfs/meta/83e4890651d57696/meta/8043419b96bfa5a5/metabase.db : success /srv/frostfs/meta/4721dcb842043023/meta/105342b45e31a08c/metabase.db : success /srv/frostfs/meta/83e4890651d57696/meta/17958f8141797034/metabase.db : success /srv/frostfs/meta/83e4890651d57696/meta/589b7712093f955a/metabase.db : success /srv/frostfs/meta/83e4890651d57696/meta/faa6163462a5f0d7/metabase.db : success /srv/frostfs/meta/4721dcb842043023/meta/3414e816b99b546c/metabase.db : success /srv/frostfs/meta/4721dcb842043023/meta/77062915fcf42434/metabase.db : success /srv/frostfs/meta/4721dcb842043023/meta/b738bb49e42766eb/metabase.db : success ```
dstepanov-yadro added 1 commit 2024-08-26 15:32:55 +00:00
[#9999] metabase: Add upgrade from v2 to v3
Some checks failed
Tests and linters / Lint (pull_request) Failing after 1m41s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m24s
DCO action / DCO (pull_request) Successful in 3m16s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m37s
Tests and linters / Run gofumpt (pull_request) Successful in 3m57s
Vulncheck / Vulncheck (pull_request) Successful in 4m21s
Tests and linters / Tests (1.23) (pull_request) Successful in 4m37s
Tests and linters / Staticcheck (pull_request) Successful in 4m31s
Build / Build Components (1.23) (pull_request) Successful in 4m40s
Build / Build Components (1.22) (pull_request) Successful in 4m41s
Tests and linters / Tests with -race (pull_request) Successful in 4m53s
Tests and linters / gopls check (pull_request) Successful in 5m30s
70db0b800c
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro force-pushed feat/metabase_upgrade from 70db0b800c to 41e4dbbf14 2024-08-26 15:33:51 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from 41e4dbbf14 to 47dc36fb44 2024-08-27 08:44:43 +00:00 Compare
dstepanov-yadro requested review from storage-core-committers 2024-08-27 08:49:25 +00:00
dstepanov-yadro requested review from storage-core-developers 2024-08-27 08:49:25 +00:00
dstepanov-yadro changed title from WIP: metabase: Add upgrade from v2 to v3 to metabase: Add upgrade from v2 to v3 2024-08-27 08:49:31 +00:00
dstepanov-yadro force-pushed feat/metabase_upgrade from 47dc36fb44 to c159736e62 2024-08-27 08:50:53 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from c159736e62 to 60ae1335fa 2024-08-27 08:51:16 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from 60ae1335fa to 3c2a572694 2024-08-27 09:01:04 +00:00 Compare
acid-ant reviewed 2024-08-27 09:21:59 +00:00
@ -0,0 +47,4 @@
}
updater, found := updates[version]
if !found {
return fmt.Errorf("unsupported version %d: no update available", version)
Member

Why we need to fail here, maybe just print debug message that update is not necessary?

Why we need to fail here, maybe just print debug message that update is not necessary?
Author
Member

Because this means that database schema is not supported by code. For example, metabase v1 will not work with v0.42 frostfs-node version.

Because this means that database schema is not supported by code. For example, metabase v1 will not work with v0.42 frostfs-node version.
acid-ant marked this conversation as resolved
@ -0,0 +68,4 @@
return fmt.Errorf("can't open new metabase to compact: %w", err)
}
if err := bbolt.Compact(dst, db.boltDB, 256<<20); err != nil {
return fmt.Errorf("failed to compact metabase: %w", errors.Join(err, dst.Close(), os.Remove(tmpFileName)))
Member

Looks like we need to remove all files which matches pattern db.info.Path + "." + "*", not only tmpFileName.
Or how about to do cleanup before starting update?

Looks like we need to remove all files which matches pattern `db.info.Path + "." + "*"`, not only `tmpFileName`. Or how about to do cleanup before starting update?
Author
Member

I suppose to not to delete files by pattern to not to delete something needed (manual backup for example)

I suppose to not to delete files by pattern to not to delete something needed (manual backup for example)
Member

That will lead to increasing garbage inside db.info.Path when node killed.

That will lead to increasing garbage inside `db.info.Path` when node killed.
acid-ant marked this conversation as resolved
dstepanov-yadro force-pushed feat/metabase_upgrade from 3c2a572694 to ed8993fc7e 2024-08-27 12:54:03 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from ed8993fc7e to 7a36729dcd 2024-08-27 13:26:33 +00:00 Compare
acid-ant approved these changes 2024-08-28 06:52:24 +00:00
dstepanov-yadro force-pushed feat/metabase_upgrade from 7a36729dcd to 2f8997a6d3 2024-08-28 11:31:10 +00:00 Compare
dstepanov-yadro changed title from metabase: Add upgrade from v2 to v3 to WIP: metabase: Add upgrade from v2 to v3 2024-08-28 13:30:32 +00:00
dstepanov-yadro force-pushed feat/metabase_upgrade from 2f8997a6d3 to 17c8967057 2024-08-28 15:33:12 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from 17c8967057 to 2d61c61bd7 2024-08-28 18:56:44 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from 2d61c61bd7 to fde016db0b 2024-08-28 19:00:20 +00:00 Compare
dstepanov-yadro changed title from WIP: metabase: Add upgrade from v2 to v3 to metabase: Add upgrade from v2 to v3 2024-08-28 19:00:25 +00:00
dstepanov-yadro force-pushed feat/metabase_upgrade from fde016db0b to 4f85d7857e 2024-08-28 19:08:44 +00:00 Compare
aarifullin approved these changes 2024-08-29 07:41:00 +00:00
fyrchik requested changes 2024-08-29 08:46:27 +00:00
fyrchik left a comment
Owner

Also, about idempotence and partial upgrades.
If the metabase is left in the inconsistent state, do we report to the user about unfinished upgrade?

Also, about idempotence and partial upgrades. If the metabase is left in the inconsistent state, do we report to the user about unfinished upgrade?
@ -0,0 +26,4 @@
path, _ := cmd.Flags().GetString(pathFlag)
noCompact, _ := cmd.Flags().GetBool(noCompactFlag)
if _, err := os.Stat(path); err != nil {
return fmt.Errorf("failed to check file: %w", err)
Owner

What did you want to check?
Doesn't meta.Upgrade return error on missing file?

What did you want to check? Doesn't `meta.Upgrade` return error on missing file?
Author
Member

moved to meta.Upgrade

moved to `meta.Upgrade`
Owner

Ah, I understand now.
If it doesn't exist, then Upgrade will create a new file, because we open with RW and this is not what we want, right?

Ah, I understand now. If it doesn't exist, then Upgrade will create a new file, because we open with RW and this is not what we want, right?
Author
Member

Exactly!

Exactly!
@ -189,6 +189,7 @@ The following table describes configuration for each shard.
| `mode` | `string` | `read-write` | Shard Mode.<br/>Possible values: `read-write`, `read-only`, `degraded`, `degraded-read-only`, `disabled` |
| `resync_metabase` | `bool` | `false` | Flag to enable metabase resync on start. |
| `resync_metabase_worker_count` | `int` | `1000` | Count of concurrent workers to resync metabase. |
| `skip_metabase_compact_on_upgrade` | `bool` | `false` | If `true` then metabase will not be compacted on upgrade. |
Owner

This is such a rarely used flag, that I do not know whether it makes any sense.
What is the tradeoff we get while changing it?

This is such a rarely used flag, that I do not know whether it makes any sense. What is the tradeoff we get while changing it?
Author
Member

Outdated

Outdated
fyrchik marked this conversation as resolved
@ -0,0 +19,4 @@
)
const (
logEveryCount = 50_000
Owner

logFrequency?

`logFrequency`?
Author
Member

ok, done

ok, done
fyrchik marked this conversation as resolved
@ -0,0 +30,4 @@
func Upgrade(ctx context.Context, path string, compact bool, log func(msg string)) error {
db, err := bbolt.Open(path, os.ModePerm, bbolt.DefaultOptions)
if err != nil {
return fmt.Errorf("failed to open metabase: %w", err)
Owner

It will be failet to .. : failed to .. : failed to ..
How about open metabase: %w?

It will be `failet to .. : failed to .. : failed to ..` How about `open metabase: %w`?
Author
Member

ok, fixed

ok, fixed
fyrchik marked this conversation as resolved
@ -0,0 +71,4 @@
if err != nil {
return fmt.Errorf("can't open new metabase to compact: %w", err)
}
if err := bbolt.Compact(dst, db, 256<<20); err != nil {
Owner

Magic constant.

Magic constant.
Author
Member

fixed

fixed
fyrchik marked this conversation as resolved
@ -0,0 +130,4 @@
case <-ctx.Done():
return ctx.Err()
case obj := <-objects:
return db.Update(func(tx *bbolt.Tx) error {
Owner

Why parallelize but use Update instead of Batch?

Why parallelize but use `Update` instead of `Batch`?
Author
Member

Outdated

Outdated
fyrchik marked this conversation as resolved
@ -0,0 +137,4 @@
if err := db.Batch(func(tx *bbolt.Tx) error {
if err := putUniqueIndexItem(tx, namedBucketItem{
name: expEpochToObjectBucketName,
key: expirationEpochKey(obj.expirationEpoch, obj.containerID, obj.objectID),
Owner

Do we always put by some new key?
Otherwise we might not behave idempotently, this is a must for upgrade.

Do we always `put` by some new key? Otherwise we might not behave idempotently, this is a must for upgrade.
Author
Member

For same object expirationEpochKey will produce the same key. Why this leads to idempotence lost?

For same object `expirationEpochKey` will produce the same key. Why this leads to idempotence lost?
fyrchik marked this conversation as resolved
@ -0,0 +164,4 @@
log("expiration epoch buckets completed completed with error: " + err.Error())
return err
}
log(fmt.Sprintf("filling expiration epoch buckets completed successfully, total %d objects", count.Load()))
Owner

Can log have signature similar to fmt.Println to avoid fmt.Sprintf on callsites?

Can log have signature similar to `fmt.Println` to avoid `fmt.Sprintf` on callsites?
Author
Member

done

done
fyrchik marked this conversation as resolved
@ -0,0 +225,4 @@
continue
}
attributeKey := string(attrKey[1+cidSize:])
if attributeKey != objectV2.SysAttributeExpEpochNeoFS && attributeKey != objectV2.SysAttributeExpEpoch {
Owner

If we need only 2 attributes, we can have 2 separate Seek loops.
It seems we currently are iterating over all attributes.

If we need only 2 attributes, we can have 2 separate `Seek` loops. It seems we currently are iterating over all attributes.
Author
Member

How? User attributes buckets have names with pattern {prefix}{cid}{attribute}.

How? User attributes buckets have names with pattern `{prefix}{cid}{attribute}`.
Owner

Oh, I see the problem -- we need to iterate specific attribute, but in all containers.

Still possible with seeks (remember current seed and seek for each container), but your implementation is more readable.

Oh, I see the problem -- we need to iterate specific attribute, but in all containers. Still possible with seeks (remember current seed and seek for each container), but your implementation is more readable.
fyrchik marked this conversation as resolved
@ -0,0 +279,4 @@
if err := objectID.Decode(attrKeyValueItem); err != nil {
return fmt.Errorf("failed to decode object id from container '%s' expiration epoch %d: %w", containerID, expirationEpoch, err)
}
it.lastAttributeKey = bytes.Clone(attrKey)
Owner

Why do we clone them? They seem to be used only during tx.

Why do we clone them? They seem to be used only during tx.
Author
Member

No, it used outside transaction:

	it := &objectsWithExpirationEpochBatchIterator{
		lastAttributeKey: usrAttrPrefix,
	}
	for {
		if err := getNextObjectsWithExpirationEpochBatch(ctx, db, it, batchSize); err != nil {
			return err
		}
No, `it` used outside transaction: ``` it := &objectsWithExpirationEpochBatchIterator{ lastAttributeKey: usrAttrPrefix, } for { if err := getNextObjectsWithExpirationEpochBatch(ctx, db, it, batchSize); err != nil { return err } ```
fyrchik marked this conversation as resolved
@ -0,0 +61,4 @@
t.Skip("for generating db")
// This test generates a metabase file with 2 million objects for 10 000 containers,
// of which
// 500 000 are simple objects,
Owner

Could you move these numbers to constants and define here instead of the comment?
It might be useful to easily adjust the test, instead of searching the whole function and thinking whether THIS 100_000 is the one I need.

Could you move these numbers to constants and define here instead of the comment? It might be useful to easily adjust the test, instead of searching the whole function and thinking whether THIS 100_000 is the one I need.
Author
Member

done

done
fyrchik marked this conversation as resolved
@ -0,0 +68,4 @@
// 100 000 million are locked (total 200 000).
db := New(WithPath(upgradeFilePath), WithEpochState(epochState{e: 1000}), WithLogger(test.NewLogger(t)),
WithMaxBatchDelay(100*time.Millisecond), WithMaxBatchSize(1000))
Owner

Do you have any specific motivation to use sth different from defaults in tests?

Do you have any specific motivation to use sth different from defaults in tests?
Author
Member

no, debugee

no, debugee
fyrchik marked this conversation as resolved
@ -0,0 +70,4 @@
db := New(WithPath(upgradeFilePath), WithEpochState(epochState{e: 1000}), WithLogger(test.NewLogger(t)),
WithMaxBatchDelay(100*time.Millisecond), WithMaxBatchSize(1000))
require.NoError(t, db.Open(context.Background(), mode.ReadWrite))
db.boltDB.AllocSize = 128 << 20
Owner

Magic constant.

Magic constant.
Author
Member

fixed

fixed
fyrchik marked this conversation as resolved
@ -0,0 +190,4 @@
lock := testutil.GenerateObjectWithCID(containers[i%len(containers)])
lock.SetType(objectSDK.TypeLock)
testutil.AddAttribute(lock, objectV2.SysAttributeExpEpoch, strconv.FormatUint(uint64(i%1000+1000), 10))
_, err = db.Put(ctx, PutPrm{
Owner

Isn't this broken? db.Put uses v3 logic, because that is what we want in our codebase and the test need to generate v2 db.

Isn't this broken? `db.Put` uses v3 logic, because that is what we want in our codebase and the test need to generate v2 db.
Author
Member

I used this test on support/v0.42 branch.

I used this test on `support/v0.42` branch.
fyrchik marked this conversation as resolved
Author
Member

Also, about idempotence and partial upgrades.
If the metabase is left in the inconsistent state, do we report to the user about unfinished upgrade?

No.
frostfs-node of current version will not work with v2 metabase.
frostfs-node of previous version will start with partially updated version, because it is impossible to add check to previous version.

> Also, about idempotence and partial upgrades. > If the metabase is left in the inconsistent state, do we report to the user about unfinished upgrade? No. `frostfs-node` of current version will not work with v2 metabase. `frostfs-node` of previous version will start with partially updated version, because it is impossible to add check to previous version.
dstepanov-yadro force-pushed feat/metabase_upgrade from 4f85d7857e to 07d1a5e6bd 2024-08-29 10:01:19 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from 07d1a5e6bd to 4bce19c1d6 2024-08-29 10:08:03 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from 4bce19c1d6 to 1bd1a6c021 2024-08-29 10:08:56 +00:00 Compare
dstepanov-yadro added 1 commit 2024-08-29 13:35:22 +00:00
[#1334] metabase: Store upgrade flag
All checks were successful
DCO action / DCO (pull_request) Successful in 3m9s
Tests and linters / Run gofumpt (pull_request) Successful in 2m55s
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Build / Build Components (1.23) (pull_request) Successful in 4m7s
Build / Build Components (1.22) (pull_request) Successful in 4m10s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m5s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m10s
Tests and linters / Tests (1.23) (pull_request) Successful in 4m17s
Tests and linters / Staticcheck (pull_request) Successful in 4m24s
Tests and linters / Tests with -race (pull_request) Successful in 5m1s
Tests and linters / gopls check (pull_request) Successful in 5m1s
Tests and linters / Lint (pull_request) Successful in 5m12s
43362f3577
This allows to check if metabase upgrade was not completed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro force-pushed feat/metabase_upgrade from 43362f3577 to dbea2c275e 2024-08-29 15:01:32 +00:00 Compare
dstepanov-yadro force-pushed feat/metabase_upgrade from dbea2c275e to 882c068410 2024-08-30 06:06:41 +00:00 Compare
fyrchik approved these changes 2024-08-30 07:26:46 +00:00
fyrchik reviewed 2024-08-30 07:29:40 +00:00
@ -0,0 +22,4 @@
upgradeLogFrequency = 50_000
upgradeWorkersCount = 1_000
compactMaxTxSize = 256 << 20
upgradeTimeout = 1 * time.Second
Owner

We use 100ms in all other code, but no big deal.

We use 100ms in all other code, but no big deal.
fyrchik approved these changes 2024-08-30 07:29:43 +00:00
acid-ant approved these changes 2024-08-30 07:55:14 +00:00
fyrchik referenced this pull request from a commit 2024-08-30 07:55:51 +00:00
fyrchik merged commit 882c068410 into master 2024-08-30 07:55:52 +00:00
aarifullin approved these changes 2024-08-30 07:59:15 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#1334
No description provided.