If all directory information is in the directory per se, then access control information must be there too.
If the file has a separate header, then access control information can be stored either in the directory entry pointing to the header (in which case different users can have different accesses) or in the file header. The latter approach is more secure, since the user in some sense "owns" his directory.