LibGDX: How to make something like vector rotation sensor with accelerometer and compass values?

You can get the rotation matrix by calling Gdx.input.getRotation(float[] matrix), it gives you the rotation matrix as per the Android SensorManager getRotationMatrix call (SensorManager.getRotationMatrix).

From that you can get a Quaternion out that is a direction and a rotation around that direction.

So if you you're looking to get the pitch, roll and yaw you can retrieve those from the Quaternion directly;

    public class MyAndroidStudioSandboxGame extends Game {

        @Override
        public void create () {
        }

        @Override
        public void render () {
            Gdx.gl.glClearColor(1, 0, 0, 1);
            Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);

            float[] mat = new float[4 * 4];
            Gdx.input.getRotationMatrix(mat);

            Matrix4 m = new Matrix4(mat);

            Quaternion q = m.getRotation(new Quaternion());

            Gdx.app.log("SANDBOX", String.format(
                "Pitch=%.2f, Roll=%.2f, Yaw=%.2f", 
                q.getPitch(), q.getRoll(), q.getYaw()));
        }
    }

The above app will log the orientation (Euler angles) of the device to logcat. There are also getPitchRad, getRollRad and getYawRad that gives you the values in radians.