IMPORTANT POINT: * whole file caching, good? bad? - remember: in AFS read and write semantics across workstation is IMPOSSIBLE (while the file is open.) **ANDREW FILE SYSTEM** ====================== - Remzi likes to ask: a sequence of call, what is state of file server/client - Security/locking in AFS **Revised NOTES** ================= # 1. Introduction Architecture: App Venus | AFS Sever (vice) | ^ ^ | --|--> v | | v <-|--- ------------------- | OS - Vice, can be set of server - Location-transparent namespace (compare to that of NFS?) - Venus: user-level + caches file from Vice, + store modified copies of files back on the servers they came from + contacts Vice only when file is opened or closed - Goal: focus on scale + because large scale degrade performance + large scale complicate administration - How: + Venus do most of the work (write locally, flush on close, etc...) + Vice work on integrity, availability, and security of file system # 2. AFS 1 - Vice - Venus interface name filed by their full pathname - Venus caches files, but will ask Vice if it cache is stale + each open of a file has at least one interaction with server Problem: not scale, * lot interaction between client and server: for cache *validity checking* * severs need to traverse full pathnames * lot of context switch between many process Solution: * reduce number of cache validity checking: callback * client/workstation do pathname traversal, not server * reduce number of server process, balance server usage by reassigning users # 3. Change for performance (actually, this is for scale) Four point: - cache management - name resolution - communication and server process structure - low-level storage representation cache management ---------------- - Server guarantee to tell client if the client's cache is invalid. - reduce # cache validation requests - when client reboot, all caches file and directory need to be validated. - Tradeoff: complexity need to maintain callback state information + before update a file or directory, server has to notify clients having callback on that file + if amount of callback state excessive: performance degrade + potential of inconsistency callback of a server may out of sync with state in other server - Hence, AFS trade complexity for scale This philosophy is different from NFS: trade simplicity for fast recovery (Hence high availability) Note: NFS is stateless, hence simple, not for scale, but for fast recovery name resolution --------------- - in AFS 1, Venus is only aware of pathnames, hence Vice has to look up mapping - AFS2: each file/directory is identify by an unique fid + Venus is aware of fid, and use it to interact with Vice + Venus lookup mapping to get fid, given a pathname + Volume location database is replicated at each server + So, at least, Venus need to look up once + (In AFS1, Vice has to lookup every time) - Note: sounds like NFS communication and server process structure ------------------------------------------ - use multithread servers, AFS1 suffer from context switch overhead - use RPC as communication mechanism + exact-one semantic in case of no failure + whole-file transfer protocol with an optimized bulk transfer protocol + authenticated communication low-level storage optimization ------------------------------ - access files by inodes rather by pathname - Data access on server: + given fid (from client), lookup vnode information + get the inode number, use iopen call to read/write data Anatomy of a call in AFS: See section 3.5 ------------------------ Client: Open("/a/b/c.txt") 1) Fetch directory a (assume not on cache) - establish callback - cache a locally 2) Fetch directory b (similar to a) 3) Fetch c.txt - establish callback - cache a locally Also, need to lookup pathname to get fid, to know which server to fetch from Consistency Model ----------------- - writes to an open file on a ws are visible to all processes all that ws - once a file is closed: + changes made to it are visible to new opens anywhere in the network + already open instances of the file do NOT reflect this changes (what about callback?) - all other file operations (e.g. protection change) are visible everywhere immediately after the operation completes - Clients can perform same operation on a file concurrently - Last-writer wins: because of whole while caching # 4. Some number to show AFS2 better # 5. Comparison with a Remote-open File System Why AFS scale well? ------------------- - due to locality of file references: server load, network traffic are reduced - whole-file transfer contacts server only on open and close + (numerous) read/write traffic are transparent to server + open and close occur at much lower frequency - efficient bulk data transfer protocol - cache entire files simplify cache management: don't deal with page-levels Some drawbacks -------------- - requires local disk, hence depends on local disk + file size > local disk cannot be accessed + e.g.: i want to stream 1TB video file (AFS cannot, NFS can) - implication: not support streaming AFS vs. NFS? ------------ - Consistency Model: + AFS: whole-file caches, callback, flush on close ~ read/write is done locally, transparent to servers ~ hence, last close win + NFS: server works like a remote disks, ~ a file can be mixed if written by multiple clients simultaneously - Caching: + AFS: whole-file cache + callback ~ less interaction with servers, hence scale ~ but need callback state at server If client crashes: discard callbacks, and revalidate on use If server crashes: tell client to discard callbacks?? + NFS: client cache stat info, data ... ~ update server every 30 seconds ~ hence, interaction with server periodically, even no read/write action ~ on read/write, need to interact with server anyway - Namespace: + AFS: transparent file location facility + NFS: does not distinguish between client and server machines ~ a machine can be both NFS server and client - Workload: + AFS: don't support streaming (hey I want to watch 1TB data) + NFS: 1 byte read, streaming - Philosophy: + AFS: scale, and hence more complex, since need to track state + NFS: for fast recovery, hence, stateless, simple # 6. Changes for Operability # Remzi class NFS: less complex AFS: scalability, (but perhaps not on performance) Problem: - open file latency: + at client side, after getting some pages, can return to the client (like prefetching) (Below is just summary of remzi note, need to take notes while re-read the paper) # 0. Take way - Main CRUX: scale - How to design a file system that scale to thousands of client + callback: get rid of many validating message + whole-file caching: make client-side performance close to local performance + use file handle instead of pathname + other: multi-threaded server, global namespace, security (read this next time) # 1. AFS1 - client fetches the whole file content, and store it in local host - upon close, send the whole new file (if modified back to server) (vs. block-based protocol of NFS) - note that in every message, client include the pathname (not file handle like NFS)n Problem: - client needs to determine if cache is still valid, hence need to ask server ==> server ends up with lot of message like this - pathname look up is costly - server is single process (to avoid context of switching), but can serve a amount of client at the same time - imbalance load Hence, AFS2 # 2. AFS2 - use file handle (similar to that of NFS) - use callback, i.e, server will notify clients that a change has been made and the info that client caches is out of date ==> reduce the number of checking message - server is multi threaded - concept of volumes # 3. Cache consistency in AFS2 - thanks to whole-file caching and callbacks, cache consistency model is easy to understand and describe E.g: C1 fetch a file from server to C1 local disk, make modification C2 fetch the same file, see the old version C1 closes, fetch changed file to server Server notifies C2 to fetch new version via calls back - it is possible that 2 clients flush change on close at almost the same time to server ==> AFS policy: *last write win*, so that we never see mixed result (thanks to whole-file caching) (vs. NFS, since NFS use block-based protocol, mixed result is possible) # 4. Crash Recovery: - a bit more complicated than NFS - Client crashes: + it may miss a callback from server + hence upon reboot, it need to check with server if it local cache is still valid (rather wait for callback from server) - Server crashes: + since callbacks is in memory, hence lost + once reboot, server has no idea about which client machines has which file + hence, need to make sure all clients know about the crash, and recheck validity of its cache content Question: how to let client know if a sever crash? + Alternative 1: has server once reboots, sends a message to clients indicating that it has just crashes, and client should not trust it cache + Alternative 2: client periodically checks if server still alive (heartbeat message, for example)