You definitely have to consider that you will 3-wheel or 2-wheel a lot more than an SFA rig. So, in some cases you will need more momentum (i.e. the loud pedal) to get over obstacles, but you have to get into some pretty hairy situations to be at this point.
However, I'm the same, not really into rock crawling, more about exploring trails to get to interesting vistas and campsites. It does well with a lift to get through some tricky trails. We also have a 2003. Plenty capable and much more controlled and comfy after lifting it on a JBA lift. In fact, so far we've only lifted the front of ours on OME HD springs over top of iron man struts (the rears are only on a stack of OEM rubber isolators). This was mainly because the front crossmember (under the oilpan) kept bottoming out. I soon will pick up a set of OMEs for the rear and will try out the bilstein shocks.
I'd also suggest getting some skid plates to prevent damage as you are driving over things. One bad line and then you would have to deal with a hole in something expensive...
Check out the camping/exploring forum here on LOST too:
viewforum.php?f=72And if you haven't been on it yet, go read up on Expedition Portal (also a forum, but not brand/model-centric).
Have fun!