Can I setup users in such a way that they will only be able to access assigned tasks in a project?

Yes.

In the user profile (accessible through Administration => Users > Users), you will need to provide your user with a “Normal” access level so that he/she will only have access to assigned projects.

Then, when assigning your user to a project, proceed as follows:

1. If you want to prevent your user from accessing unassigned tasks in the project, do not grant the “Can Open This Project” right. This set up will allow the user to access only assigned and created tasks (if the user has sufficient Edit Task Level rights on the project to create tasks) in the project, through the Home tab only.
2. If you want to give read-only access to tasks in the project, choose A1-Read Only as the Edit Task Level.