AC reactor VS DC reactor in VFD

AC reactors reduce harmonics, mitigates transient conditions and helps to maintain proper phase to phase voltages. DC reactors will also mitigate harmonics however it will provide more harmonic attenuation than a equally sized AC reactor.

Both reactors will increase the conduction angle of the DC current pulses flowing through the rectifiers and thereby reduce the harmonics on the supply.
The AC line reactor will provide a high impedance path for fast transients and reduce the transient voltages across the rectifiers providing a level of transient voltage protection for the input rectifier.
Both reactors will reduce the charging current to the capacitor bank under a voltage surge condition, so help to protect the rectifier from current surges resulting from voltage surges/transients.
The AC line reactor has an AC flux in the iron, with a high harmonic content, and so has both copper loss and iron loss.
The DC bus choke has DC with a superimposed ripple current, so the iron losses are lower.

Major differences? AC has higher losses but improved transient voltage protection for the rectifier. DC has lower losses, so higher efficiency.

Very important is that most DC bus chokes will need to be custom designed for the drive as they are commonly mounted inside the drive chassis. (and frankly, I can't think of any reason to use a bus choke over a line reactor if both would have to be externally mounted, the internal mount is the only real advantage to the bus choke). Thus while there may be less weight for a bus choke, it's cost can be higher. On the other hand, multiple manufacturers have a wide range of ratings for 3 phase line reactors. To the point where the 3 phase reactor is practically a commodity and thus can be obtained very quickly and inexpensively. This is also important if a replacement is ever needed.True in most cases the reactor will need to be mounted external to the drive chassis, but this is offset by the cost savings of buying the reactor direct from a distributor. And not paying double markup necessary when it comes in the drive from the VFD OEM. Also, since the BUS choke is custom, there will be little option for varying the inductance and thus weighing cost vs performance for different applications. Reactors are commonly available in 1.5%, 3%, 5% and 10% impedance ratings.