Yes, the OS releases leaked memory on exit. At least, this is what happens when a new application starts and the OS manages memory allocation for execution.
On the other hand, Unity's Play Mode happens within the IDE itself, which is already running when Play Mode starts. This is different from other engines (such as GameMaker Studio), which compile and run your game as an independent executable every time you need it.
I suppose the OS is dealing with actual memory management for Unity. And Unity is only managing memory for Play Mode at a high level. This allows Unity to provide a useful NativeLeakDetectionMode and memory leaks detection. Allocator types include Temporary and Persistent allocation modes. Unfortunately, the related documentation is poor. One can find more information about NativeArrays on the Unity Forums, although that information will be coming from users rather than Unity devs.
In this forum thread, user recursive describes how different allocator types work. They say that Temporary allocators will trigger warnings after few frames, whereas Persistent ones give more freedoms to the developers. This is likely due to Unity monitoring memory in Play Mode.
However, the IDE can make no assumptions on memory usage when an allocator works in Persistent mode. It doesn't know if that memory is needed for 10 frames or 10 minutes. Then, it delegates control to the developers, who aren't annoyed by warning messages but are in charge of disposing of memory when not needed anymore.
Leaked memory is still leaked memory − the longer an application runs, the more it keeps leaking. That's why Unity still triggers the warning. Better use NativeArray<T0>.Dispose().