What you are seeing is pretty typical for the newer generation MEMS based heading sensors, although, more likely it would behave like Johnny pointed out and eventually settle again.
Other effects are a very nervous heading when the boat rolls or pitches and all these effects are worse at high latitudes.
In general, classic gimballed flux-gate compasses, especially when assisted by a rate giro still do much better than any of these new devices. What you have is not a flux-gate compass, but one based on MEMS chips (micro-electro-mechanical-systems) that have become very cheap, given that they are in every smart phone.
But technically it is a real challenge to combine these to get a result as good as from a gimballed and oil dampened flux-gate system. The gimbals keep the compass coils level and the oil avoids crazy swings. The short term deviations, due to slow swinging, are sorted out by the rate giro that only needs to look at the yaw axis.
The fixed mount MEMS based magnetometer will not be horizontal all the time and since the earth magnetic field, especially in higher latitudes is not at all oriented horizontal a non level sensor gets a much different reading. This gets pretty complicated to compensate, as you need the accelerometer to determine from the force of gravitation, what the orientation of the system is to then correct the magnetometer. But the accelerometer is also exposed to actual acceleration forces and can only over time average out where the center of earth is. Again that time needs to get bridged by the rate giro.
All this is very compute intensive and overly sensitive to the classic sources of compass error and I have yet to see a MEMS implementation that gets near the classic rate giro assisted flux-gate sensor in performance. I have found one that uses 3 separate analogue output accelerometers and they do the integration in analog circuitry, rather than complex computing and they seem to do much better.
This is probably the reason why Furuno sticks to proven flux-gate designs with their heading sensors, that really perform very well.