Almost certainly a compass calibration problem, yes. There is a ground calibration you need to do first (compass rose is very helpful for this) and then an in-air calibration as well after the ground calibration. If the problem persists beyond that, then you likely have some magnetic interference near the magnetometer (either internal to the ADAHRS unit, or an external magnetometer) that will need to be corrected.
The root cause here is that without a good compass calibration, the unit has no idea which direction the aircraft is actually pointed. The wind is calculated by the difference between a GPS plot and the speed/heading of the aircraft as determined by the ADAHRS, which requires knowing accurately which way the airplane is pointed. Without that, the wind calculation is garbage in, garbage out.
