sjoerdjob.com

Confusion between FILE *s and file descriptors permalink

The problem

The source was a large data-file that needed to be parsed and sent to a database. I did not feel like writing a library for it myself, so I re-used an existing command line based parser which is written in C.

The parser

Think of something like running ffmpeg on a stream, grep-ing through a file, or some similar weird action.

#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>


int main(int argc, char **argv) {
    if (argc > 1) {
        printf("Usage: %s\n", argv[0]);
        printf("This program reads from stdin.\n");
        exit(-1);
    }

    char buffer[1000];
    int amtread = 0;
    assert(buffer);

    do {
        amtread = fread(&buffer, 1, 1000, stdin);
        printf("%d\n", amtread);
        sleep(random() % 4);
    } while (amtread > 0);
}

The actual parser was a lot more complicated, and actually did something interesting with the data, but this is nicer to strace to see what's happening (because not much happens).

The initial solution? epoll?

Now, I know a lot about epoll and select, so one of the ideas that came to mind was creating a pipe, passing the read end to the child process, and using epoll to make sure the write end was always filled while also keeping a counter as to how many bytes were written into the buffer.

Now, the problem with using epoll is that it is not always as easy to use as one would like, and quite often always complex. On the other hand, it is quite powerful, but I decided to see if there was a simpler approach.

Sharing file pointers?

Another idea (which I needed to test) was seeing if the file-pointer gave some shared state. After all, they point to the same object before fork(), and maybe they remain in sync afterwards. I was doubtful, but decided to test it anyway.

But there is a bit of a problem with this. The file pointers are based on FILE *. They are chunks of data in user space memory. On fork(), the user space memory gets copied (actually: marked for copy-on-write) to create the memory for the new fork. So, updates in the child (resp. parent) are not visible in the parent (resp. child).

And how about file descriptors?

It was not obvious to me at first, but then I thought: You have file pointers and file descriptors. And there's a difference in working with them. Also, file descriptors are just integers. And they don't change. Another clue as to why this might work is that the stdout of a forked process is just as visible as that of the forking process. So there is hope for that.

Re-reading the manual page on fork gives us this gem:

FORK(2)                     BSD System Calls Manual                    FORK(2)

NAME
     fork -- create a new process

... <snip>

           o   The child process has its own copy of the parent's descriptors.
               These descriptors reference the same underlying objects, so
               that, for instance, file pointers in file objects are shared
               between the child and the parent, so that an lseek(2) on a
               descriptor in the child process can affect a subsequent read or
               write by the parent.  This descriptor copying is also used by
               the shell to establish standard input and output for newly cre-
               ated processes as well as to set up pipes.

... <snip>

Reading closely again: "... shared between the child and the parent ...". This sounds exactly like what we need. And it also gives us a bit of a clue: lseek. Traditionally, lseek is used to move the position of where to read next, but it has a very interesting return value: the resulting position (measured in bytes) from the beginning of the file.

In Python, we can do the following:

current_position = os.lseek(fp.fileno(), 0, os.SEEK_CUR)

And this, my friends, gives us exactly what we want.

Recipe

In the end, I found the following recipe quite workable:

import os
import subprocess
import time

with open('/path/to/large.file') as fp:
    fd = fp.fileno()
    file_size = os.fstat(fd).st_size
    child = subprocess.Popen(['/path/to/command'], stdin=fd)
    while child.poll() is None:
        time.sleep(1)
        current_position = os.lseek(fp.fileno(), 0, os.SEEK_CUR)
        print("Progress: {:0.2f}".format(100.0 * current_position / file_size))

About this recipe, I would like to make a final remark: this is not code how I would implement it in a final design. It is too tightly coupled. But, it is valuable code nonetheless for two reasons:

• First and foremost: It shows that a solution is possible, and the necessary ingredients thereof.

• Second: it gives a succinct and runnable example of how to implement it.

As with all code-recipes: the recipe is not valuable as a copy-and-paste blob of code, but for the ingredients it contains. The user of the recipe is responsible for mixing it to their own liking.