Since the code is tearing down the interface bits it did sort of made sense, but indeed, I may have been wrong here. Anyhow, I did ask Seb if he can have time to look on this, meanwhile I will also try to dig in more and see if I can get better understanding about this code.

The change is correct. The code should be doing (pardon the verbose pseudo-code):
if physical interface pi removed {
        for each prefix p on pi {
                if prefix is used for address autoconfiguration {
                        delete addresses created from that prefix
                }
                delete the prefix
        }
        cleanup remaining pi state
}

So as this is the fix afterall, I'll drop this issue.