You say :

Then I mul that by inverted local inertia to get angular acceleration in local space.

This seems wrong, since "inverted local inertia" is "world inertia", so you are putting "that" to world space ...

So code should be :

// oriMat: local -> world
oriMat = quaternion.ToMatrix(self.ori)

// put total torque in local space
self.totalTorque = vector.Transform(self.totalTorque, matrix.Inverted(oriMat))

// multiply local torque with local inertia to get angAcc in local space
angAcc = vector.Transform(self.totalTorque, self.inertia_local))

// put angAcc in world space
angAcc = vector.Transform(angAcc, oriMat)

You say :

Then I mul that by inverted local inertia to get angular acceleration in local space.

This seems wrong, since "inverted local inertia" is "world inertia", so you are putting "that" to world space while you want it in local space ...

So code should be :

// oriMat: local -> world
oriMat = quaternion.ToMatrix(self.ori)

// put total torque in local space
self.totalTorque = vector.Transform(self.totalTorque, matrix.Inverted(oriMat))

// multiply local torque with local inertia to get angAcc in local space
angAcc = vector.Transform(self.totalTorque, self.inertia_local))

// put angAcc in world space
angAcc = vector.Transform(angAcc, oriMat)