add RGB -> depth mapping

[floe]

As promised on the list, here's a preliminary patch to map RGB data onto the depth image instead of the other way round (FREENECT_DEPTH_REGISTERED). I haven't integrated this functionality as a video mode, since you need synchronized depth and video images for this to work. Since libfreenect doesn't do synchronization, this is left to the frontend.

[fran6co]

I found a weird bug, it's duplicating objects against far away background.

Not sure why, trying to figure it out.

[fran6co]

@floe, I found out what is causing the problem. When doing depth to rgb you could have different depths mapped to the same same rgb coordinate but you can discard the furthest one. If you just use the same mapping but inverse it, as you did, you'll have different depth coordinates mapped to the same color causing duplicated objects.
The solution would be generating a new registration table, but that part of the code is not that well documented to figure out how to generate the proper table.

[floe]

I think it's physically impossible to generate a different registration table which doesn't show this problem... in your example, the pixels where the left image of the spoon is visible can be directly seen by the depth camera, but from the point of view of the RGB camera, they are behind the spoon, so the mapping is sort-of-correct.

Generally, this mode is mostly applicable for specific applications (e.g. when you want to determine the color of a depth-segmented object and need the best depth resolution you can get, as the depth -> RGB mapping will lower the resolution a bit).

[fran6co]

But the same thing happens when doing depth->rgb, you'll have some pixels that are visible in the rgb camera but not on the depth and mapping depth to rgb doesn't show this type of artifacts.
One way to solve it, would be blacking out those pixels but it would cause a double shadow.
I think it's possible to map rgb to depth without double shadow or this kind of artifacts, projects like http://labs.manctl.com/rgbdemo/ or http://idav.ucdavis.edu/~okreylos/ResDev/Kinect/ work just fine but with manual calibration.

[fran6co]

I was wrong, you'll always get a double shadow when doing registration (one for the parallax between the IR camera and the projector and the other for the IR and the rgb). The problem is figuring out what pixels in the depth camera are not visible in the rgb, so the registration can blacken them out.
I'm trying some extra arrays to keep track of duplicated pixels, but it doesn't work that well.

[fran6co]

@floe, what do you think about a zBuffer? It fixes most of the artifacts (there a couple of artifacts that can't be saved because of the distortion)

    uint32_t target_offset = dev->registration.reg_pad_info.start_lines * DEPTH_Y_RES;
    int x,y;
    int* map = (int*)malloc(DEPTH_Y_RES*DEPTH_X_RES* sizeof(int));
    unsigned short* zBuffer = (unsigned short*)malloc(DEPTH_Y_RES*DEPTH_X_RES* sizeof(unsigned short));

    for (y = 0; y < DEPTH_Y_RES; y++) for (x = 0; x < DEPTH_X_RES; x++) {

            uint32_t index = y * DEPTH_X_RES + x;
            uint32_t cx,cy,cindex;

            map[index] = -1;

            int wz = depth_mm[index];

            if (wz == 0) {
                continue;
            }

            // coordinates in rgb image corresponding to x,y in depth image
            cx = (dev->registration.registration_table[index][0] + dev->registration.depth_to_rgb_shift[wz]) / REG_X_VAL_SCALE;
            cy =  dev->registration.registration_table[index][1] - target_offset;

            if (cx >= DEPTH_X_RES) continue;

            cindex = cy*DEPTH_X_RES+cx;
            map[index] = cindex;

            if (zBuffer[cindex] == 0 || zBuffer[cindex] > wz) {
                zBuffer[cindex] = wz;
            }
        }

    for (y = 0; y < DEPTH_Y_RES; y++) for (x = 0; x < DEPTH_X_RES; x++) {
        uint32_t index = y * DEPTH_X_RES + x;
        uint32_t cindex = map[index];

        // pixels without depth data or out of bounds are black
        if (cindex == -1) {
            index *= 3;
            rgb_registered[index+0] = 0;
            rgb_registered[index+1] = 0;
            rgb_registered[index+2] = 0;

            continue;
        }

        unsigned short currentDepth = depth_mm[index];
        unsigned short minDepth = zBuffer[cindex];

        // if the pixels are near we keep the mapping
        if (2*(currentDepth - minDepth) < (currentDepth + minDepth)) {
            index *= 3;
            cindex *= 3;

            rgb_registered[index+0] = rgb_raw[cindex+0];
            rgb_registered[index+1] = rgb_raw[cindex+1];
            rgb_registered[index+2] = rgb_raw[cindex+2];
        }
    }

    free(zBuffer);
    free(map);

[floe]

This looks promising! Can you explain this condition:

        if (2*(currentDepth - minDepth) < (currentDepth + minDepth))

[fran6co]

The rgb-depth mapping maps several points in the depth to the same rgb value. Some of them are the ones that are visible from the depth camera but not the rgb camera and some of them are just neighboring points. That condition is to separate the 2 cases.

[fran6co]

@piedar Why was this closed?

[floe]

@fran6co could you generate a new branch from my old unstable branch and integrate your patch? then you can submit a pull request based on that.

[piedar]

Sorry, it got closed when I deleted the now-unused unstable branch. I can't reopen the issue, so floe's suggestion is a good one.

[fran6co]

I'll do it during this week. Maybe today if I have the time.