Hi Divya,
The Select Expert has a GUI that lets you select fields to apply the filter. The Record Selection formula on the other hand is a code editor where you type in the filter formula.
When you add a filter using the Select Expert, it automatically creates a Record Selection Formula for you. This, however does not let you create complex selection formulas. Complex formulas are created by directly writing the code in the Record Selection Formulas editor.
-Abhilash