implement a way to deduplicate labels easily

[gdementen]

this issue is about appending some suffix (eg _1, _2) to duplicate labels, but the end goal is to make "aggregate all duplicate labels together" as easy as possible.

[gdementen]

Here is the code Bernard is currently using to aggregate all duplicate labels. We need to make this easier (and more generic obviously).

def agg_dupl(mat):
    l_c = []
    for i, p in enumerate(mat.prods):
        l_c.append((p + '_' + str(i)) if p in l_c else str(p))
    mat = mat.set_labels(x.prods, l_c)
    agg = tuple(mat.prods.startswith(code) >> code for code in mat.prods.matches('^.....$'))
    return mat.sum(agg)

Here is a first step.

class Axis(object):
    def deduplicate(self):
        # TODO: handle non-string labels (unsure how!? add a user-defined constant?)
        seen = set()
        labels = []
        for i, label in enumerate(self.labels):
            if label in seen:
                label = '{}_{}'.format(label, i)
            else:
                seen.add(label)
            labels.append(label)
        return Axis(labels, self.name)

def agg_dupl(mat):    
    mat = mat.set_axes('prods', mat.prods.deduplicate())
    agg = tuple(mat.prods.startswith(code) >> code for code in mat.prods.matches('^.....$'))
    return mat.sum(agg)

Remaining problems:

• the aggregation relies on the fact his codes are always exactly the same length
• we should provide a way to "deduplicate()" all axes at once

We should be able to write something like:

def agg_dupl(mat):
    mat = mat.deduplicate()
    agg = tuple(mat.prods.startingwith(code) >> code for code in mat.prods[~mat.prods.endswith('/DUPE/')])
    return mat.sum(agg)

or even

def agg_dupl(mat):
    mat = mat.deduplicate()
    agg = tuple(mat.prods.startingwith(code) >> code for code in ~mat.prods.endingwith('/DUPE/'))
    return mat.sum(agg)

Remaining problems:

• ideally the .deduplicate() step should not be necessary. It should be possible to easily create a positional-based agg directly.

[gdementen]

This particular problem (aggregating all duplicates) can be solved much more elegantly by a groupby.

agg = mat.groupby(mat.prods).sum()
# and/or
agg = mat.sum(mat.prods.groupunique())

The question is then whether de-duplication has value outside of aggregating the duplicates. I can see some cases where this is true, but it is lower priority I think.