Does Docker use UnionFS, and does it cause "fallocate:: Operation not supported"?

MongoDB Percona Operator for MongoDB
1

In a question that was originally opened under the forum for PSMDB (link) the following error was observed when using PSMDB 4.4 in the operator.

psmdb_1 | {"t":{"$date":"2020-09-01T19:50:00.963+00:00"},"s":"E", "c":"STORAGE", "id":22435,  "ctx":"thread1","msg":"WiredTiger error","attr":{"error":95,"message":"[1598989800:963675][1:0x7f882be42700], log-server: __posix_std_fallocate, 58: /data/db/journal/WiredTigerTmplog.0000000003: fallocate:: Operation not supported"}}
psmdb_1 | {"t":{"$date":"2020-09-01T19:50:00.963+00:00"},"s":"E", "c":"STORAGE", "id":22435,  "ctx":"thread1","msg":"WiredTiger error","attr":{"error":95,"message":"[1598989800:963758][1:0x7f882be42700], log-server: __posix_sys_fallocate, 75: /data/db/journal/WiredTigerTmplog.0000000003: fallocate:: Operation not supported"}}

As the error message shows the unix syscall fallocate() simply failed. MongoDB might retry a few times, but if can’t get file space allocated it has to give up, and error-exits.

The following Stackoverflow QA suggests the syscall failure is a limitation of docker: https://stackoverflow.com/questions/31155591/fallocate-failing-inside-my-docker-container .

Is this a true problem, and does it affect PSMDB operator now?

If this isn’t the cause then I guess fallocate() is failing because there truly is no filesystem space to allocate. How would we debug that in kubernetes?

2

Hello @Akira_Kurogane ,

fallocate is supported by most of the filesystems. For MongoDB case specifically it depends where you store the data:

For example if you run MongoDB in docker without specifying any storage dirs - data is stored on default docker storage driver. In ubuntu it is overlay2. I tested fallocate - worked fine for me.

I replied in the original thread as well - would be great to see docker info first and understand where data is stored, what is the filesystem used.

For PSMDB Operator - we should be fine. It still depends where data is stored. But for most of the cases (AWS or GCP) Persistent Volume Claims use battle-tested filesystems like ext4 or xfs. They do support fallocate.

3

Hi Sergey. Thanks for trying out the kubernetes environment part, that’s what I don’t know.

> I tested fallocate [in an overlay2 filesystem-using case] - worked fine for me.

Thanks. So I guess we should rule out that it was just overlay2 fs alone. At most, it might be overlay2 filesystem in combination with transient resource failure such as a lack of memory.

Or … maybe it was overlay 1-not-2. I’ll ask the community user Stef for the docker info as well.