I just read the strategy guide again and found out I missed some stuff while I was exploring!
How I left the shrine is to go south and head to any direction.
As for leaving the desert, the quickest way is by the northwest path, south of the cave where I found the Sand Cape. I burrow below and leave the desert behind!
I go back to the upper part of the mine, and go to the water where I had to fight those blue slimes.
Turns out the guide said that there were 50 of them I had to fight!
After getting all 50 of them, which felt like an eternity (but only 8-10 minutes, I think), I get a chest containing herbs.
I head south, where the iron bars are now gone and I end up falling down the east house's chimney to reach a chest containing a Gilded Falcon. I push the lever opening the gate.
I explore more of the area, heading southwest past the Pond. At the west, I'm near a place called the Fire Manor.
I go north to find tall pillars and an orc with a swinging flail!
After grabbing a chest of herbs, I double back.
I bomb the stumps and head east, and I'm now near Nava's house. I bomb another stump north to get a chest with a Gilded Falcon inside. I now have 11 of these. What are these falcons for anyway?
I break the square block westward and cross north to find a chest with a Magic Seed inside!!!
This means my magic meter has upgraded to 2!
Since I have lots of gilders, I go to the Cliffs to buy some more stuff from Lurvy!
I get a Life Vessel, which costs 400 gilders, upgrading my health to 16!!
I climb the Cliffs again to get one more chest...30 gilders.
Now I head back to the shrine after that long detour!