You say "This may be causing my site problems?"... but what problems are you actually having? Is your site crashing? Is it loading slowly? How long have these problems been happening? Are they getting worse over time? Are the problems across all pages on your site or more noticable on particular pages?
Specifically regarding the memory limit: If you have a problem with the memory limit, it will mean that your site would crash when it hits the limit. Before the limit, it will work fine, so if the problem is loading speed, then changing the memory limit won't affect that. You can tell if memory limit is a problem by looking in your server logs for out-of-memory errors.
There's a whole bunch of other stuff that you might be able to do to performance tune your system, but you'd need to give more detail about exactly what the problem is before I can really help any more than that.
However, one thing I can say: There is no way that a single PHP program should need 3 gig of memory of its own! Even 523 meg is higher than I would normally expect. A PHP program should use a minimum amount of memory possible because it's supposed to load and finish in a short time. If it takes longer and uses more memory, then it can severely affect the number of concurrent users your system can support.
I suggest doing some diagnostics to find out what the real bottleneck is in your system before trying to fix it by arbitrarily increasing the memory limit.
Hope that helps.
