I wouldn't worry about the system using swap while there is plenty of free RAM available. As others have stated, it is a rather common situation. The kernel might decide on moving some seldom accessed memory pages to swap in order to make RAM available for future demand. But when RAM starts running low, do keep and eye on how many bytes are actually being swapped. You can use vmstat to show the amount of bytes being swapped in/out of the system.
I.e: vmstat output of a system with no swapping taking place and marginal swap usage:
~# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 28052 363992 156736 1251116 0 0 6 15 1 7 2 1 97 0
0 0 28052 363964 156736 1251128 0 0 0 13 759 283 2 2 96 0
0 0 28052 371132 156736 1251132 0 0 0 5 348 287 1 1 98 0
Any value > 0 means the system is actually reading from or writing to swap, at the same time you should notice a severe downgrade of the system's performance.