Recently I noticed that the service vmmem had high memory usage at the same time I was not running any containers in Docker Desktop, the memory usage was actually around 96%, the CPU usage was higher than normal as well. The key was to create a .wslconfig file (windows) in your %USERPROFILE% folder with the config options from the Microsoft Site. Below is what I am currently using.
memory=6GB # limits VM memory in WSL to 6 GB only
processors=4 # limits the WSL2 to use 4 virtual processors only
Once you create the file and add the options you can restart the LX subsystem using the following Powershell command (must be at an elevated prompt)
You may get some errors in Docker Desktop about communication but you can just restart Docker Desktop to get rid of them.