Deleted container sporadically appears in the container list immediately after deletion

an-nikitin commented

2023-07-10 13:26:55 +00:00

When a container is deleted with "frostfs-cli container delete" and then "frostfs-cli container list" is invoked immediately after, the container is sometimes (with the probability of around 30%) present in the list. It goes away in several seconds, presumably after the morph.cache_ttl interval expires.

Expected Behavior

The deleted container is never present in the container list.

Current Behavior

The deleted container is sometimes (with the probability of around 30%) present in the list and goes away in several seconds, presumably after the morph.cache_ttl interval expires.

Possible Solution

No fix can be suggested by a QA engineer. Further solutions shall be up to developers.

Steps to Reproduce (for bugs)

Have the cluster ready for storage operations.
Run "tatlin-object-check StorageBasicFunctions"; it should report that the deleted container is found in the container list; if it doesn't, repeat several times and the problem is likely to manifest itself.
Since the check uses frostfs-cli commands, performing the container create, delete and list commands manually should cause the problem to manifest too.

Context

This issue causes the tatlin-object-check command (used to check the cluster health) to fail sporadically.

Regression

No.

Your Environment

A SberCloud cluster, T.O 1.3.0-41.

When a container is deleted with "frostfs-cli container delete" and then "frostfs-cli container list" is invoked immediately after, the container is sometimes (with the probability of around 30%) present in the list. It goes away in several seconds, presumably after the morph.cache_ttl interval expires. ## Expected Behavior The deleted container is never present in the container list. ## Current Behavior The deleted container is sometimes (with the probability of around 30%) present in the list and goes away in several seconds, presumably after the morph.cache_ttl interval expires. ## Possible Solution No fix can be suggested by a QA engineer. Further solutions shall be up to developers. ## Steps to Reproduce (for bugs) 1. Have the cluster ready for storage operations. 2. Run "tatlin-object-check StorageBasicFunctions"; it should report that the deleted container is found in the container list; if it doesn't, repeat several times and the problem is likely to manifest itself. Since the check uses frostfs-cli commands, performing the container create, delete and list commands manually should cause the problem to manifest too. ## Context This issue causes the tatlin-object-check command (used to check the cluster health) to fail sporadically. ## Regression No. ## Your Environment A SberCloud cluster, T.O 1.3.0-41.

an-nikitin added the

bug

triage

labels 2023-07-10 13:26:55 +00:00

fyrchik added this to the v0.37.0 milestone 2023-07-10 13:27:33 +00:00

fyrchik added

frostfs-node

and removed

triage

labels 2023-07-10 13:28:27 +00:00

ale64bit self-assigned this 2023-07-19 06:58:28 +00:00

ale64bit commented

2023-07-24 11:18:39 +00:00

Member

I'm not able to repro this. I've tried in dev-env with different values of epoch duration and morph ttl cache.

Here's the script I'm using:

package main

import (
	"flag"
	"log"
	"os/exec"
	"strings"
)

var (
	cli         = flag.String("cli", "/path/to/frostfs-node/bin/frostfs-cli", "")
	rpcEndpoint = flag.String("rpc_endpoint", "s01.frostfs.devenv:8080", "")
	wallet      = flag.String("wallet", "/path/to/frostfs-dev-env/wallets/wallet.json", "")
)

func main() {
	flag.Parse()

	var cid string
	{
		cmd := exec.Command(*cli,
			"container", "create",
			"--policy", "REP 1",
			"--await",
			"--rpc-endpoint", *rpcEndpoint,
			"--wallet", *wallet)
		out, err := cmd.CombinedOutput()
		if err != nil {
			log.Fatalf("creating container: %v\n%s\n", err, string(out))
		}
		cid = strings.TrimSpace(strings.TrimPrefix(strings.Split(string(out), "\n")[0], "container ID: "))
		log.Printf("container created: cid = %q", cid)
	}

	{
		cmd := exec.Command(*cli,
			"container", "list",
			"--rpc-endpoint", *rpcEndpoint,
			"--wallet", *wallet)
		out, err := cmd.CombinedOutput()
		if err != nil {
			log.Fatalf("listing containers: %v", err)
		}
		s := strings.TrimSpace(string(out))
		log.Printf("list output: %q", s)
	}

	{
		cmd := exec.Command(*cli,
			"container", "delete",
			"--cid", cid,
			"--await",
			"--rpc-endpoint", *rpcEndpoint,
			"--wallet", *wallet)
		if out, err := cmd.CombinedOutput(); err != nil {
			log.Fatalf("deleting container: %v\n%s\n", err, string(out))
		}
		log.Println("container deleted")
	}

	for {
		cmd := exec.Command(*cli,
			"container", "list",
			"--rpc-endpoint", *rpcEndpoint,
			"--wallet", *wallet)
		out, err := cmd.CombinedOutput()
		if err != nil {
			log.Fatalf("listing containers: %v", err)
		}
		s := strings.TrimSpace(string(out))
		log.Printf("list output: %q", s)
	}
}

I'm not able to repro this. I've tried in dev-env with different values of epoch duration and morph ttl cache. Here's the script I'm using: ```go package main import ( "flag" "log" "os/exec" "strings" ) var ( cli = flag.String("cli", "/path/to/frostfs-node/bin/frostfs-cli", "") rpcEndpoint = flag.String("rpc_endpoint", "s01.frostfs.devenv:8080", "") wallet = flag.String("wallet", "/path/to/frostfs-dev-env/wallets/wallet.json", "") ) func main() { flag.Parse() var cid string { cmd := exec.Command(*cli, "container", "create", "--policy", "REP 1", "--await", "--rpc-endpoint", *rpcEndpoint, "--wallet", *wallet) out, err := cmd.CombinedOutput() if err != nil { log.Fatalf("creating container: %v\n%s\n", err, string(out)) } cid = strings.TrimSpace(strings.TrimPrefix(strings.Split(string(out), "\n")[0], "container ID: ")) log.Printf("container created: cid = %q", cid) } { cmd := exec.Command(*cli, "container", "list", "--rpc-endpoint", *rpcEndpoint, "--wallet", *wallet) out, err := cmd.CombinedOutput() if err != nil { log.Fatalf("listing containers: %v", err) } s := strings.TrimSpace(string(out)) log.Printf("list output: %q", s) } { cmd := exec.Command(*cli, "container", "delete", "--cid", cid, "--await", "--rpc-endpoint", *rpcEndpoint, "--wallet", *wallet) if out, err := cmd.CombinedOutput(); err != nil { log.Fatalf("deleting container: %v\n%s\n", err, string(out)) } log.Println("container deleted") } for { cmd := exec.Command(*cli, "container", "list", "--rpc-endpoint", *rpcEndpoint, "--wallet", *wallet) out, err := cmd.CombinedOutput() if err != nil { log.Fatalf("listing containers: %v", err) } s := strings.TrimSpace(string(out)) log.Printf("list output: %q", s) } } ```

fyrchik modified the milestone from v0.37.0 to v0.38.0

2023-08-29 09:32:39 +00:00

fyrchik referenced this issue

2023-11-14 11:09:04 +00:00

frostfs-cli command 'container list' returns just deleted container #801

fyrchik referenced this issue from a pull request that will close it,

2023-11-14 11:41:49 +00:00

Invalidate list cache on container events #802

fyrchik closed this issue

2023-11-14 19:00:14 +00:00

fyrchik referenced this issue from a commit

2023-11-14 19:00:15 +00:00