Recently I noticed that the service vmmem had high memory usage at the same time I was not running any containers in Docker Desktop, the memory usage was actually around 96%, the CPU usage was higher than normal as well. The key was to create a .wslconfig file (windows) in your %USERPROFILE% folder with the config options from the Microsoft Site. Below is what I am currently using.
[wsl2]
memory=6GB # limits VM memory in WSL to 6 GB only
processors=4 # limits the WSL2 to use 4 virtual processors only
Once you create the file and add the options you can restart the LX subsystem using the following Powershell command (must be at an elevated prompt)
Restart-Service LxssManager
You may get some errors in Docker Desktop about communication but you can just restart Docker Desktop to get rid of them.