CS 736 Reviews - Spring 2015

1.Summary
The paper discusses a new approach in software development for multicore processors using scalable commutativity. The programming interfaces are analyzed by identifying the operations that commute and allow the interfaces to executed in any non-conflicatble order. A tool called COMMUTER is introduced for programmers to analyze the interface commutativity, generate commutative operations and test a scalable implementation. The idea has been tested on sv6 OS POSIX calls and some performance measurements on test cases are given.

2.Problem
Scalability of a software for multicore processors is generally tackled as an implementation problem. Once the software is implemented and tested on various benchmarks, you can discover bottlenecks of the interface and related environment which could limit the scalability. Authors suggest that fixing the software at that stage of the design is a problem and propose a solution to discover interface’s scalability during implementation stage and attribute to software scalability.

3.Contributions
The important contribution of paper is the introduction of a commutativity called state-independent, interface-based and monotonic (SIM). They show that when operations commute with this commutativity rule in a system, an implementation can exist which can execute without any conflicts and thus achieve more concurrency without explicit synchronization or communication. This can be applied to real interfaces and thus achieve software scalability. The paper also describes possible scalable interfaces for a real system with POSIX calls and techniques such as non-determinism and freeing resources at a later stage. The authors also have built a tool — COMMUTER to automate the interface analyzing, generate commutative operations for test cases and test the implementation as conflict-free and scalable. These 3 phases of the tool are called ANALYZER, TESTGEN and MTRACE respectively.

4.Evaluation
COMMUTER has been evaluated for sv6’s scalability by modeling several POSIX file system and Virtual memory calls in Commuter. Test cases have been listed out and compared with Linux’s scalability. Experiments are run with sv6 on a 80-core machine to check the performance and scalability of commutative operations on two micro-benchmarks and one application benchmark. Results suggest that sv6 scales for 99% test cases whereas Linux with regular interfaces was only scalable upto 68% of cases.

5.Confusion
More clarification on monotonic and non-monotonic requirement of SIM commutativity will be helpful. What could be the system storage/state overhead for sv6 interfaces for an 80-core machine as multiple conflict-free implementations could exist?

1. Summary
The authors present the novel idea that when designing scalable systems, the interfaces can be examined for commutativity to determine scaling-ability. The SIM-commutativity rule is formalized, common scaling issues are examined, and the COMMUTER system which allows for analysis of symbolic interfaces is presented.

2. Problem
Designing scalable systems is hard. To find issues developers typically pick a workload, scale the number of cores, and find bottlenecks. As a result developers can’t easily ascertain whether these bottlenecks are fundamental to the design, or simple implementation issues. It is often too late to fix the issues that are fundamental to the design. Instead, the authors want to move the idea of scalability to the software interface, where developers can quickly find scaling problems before they are baked in to the final implementation. Previous works have used the idea of commutativity for concurrency safety, but have not considered scalability.

3. Contributions
The main contribution of the paper is the SIM-commutativity rule. The authors found that the simple algebraic definition of commutativity was too stringent. Instead, they created a definition that says if a history Y SIM-commutes in H, there exists a correct implementation in which the region Y is conflict free. This means that memory accesses are to mutually exclusive regions of cache lines, meaning that the interface scales. SIM-commutativity in particular means that an operation must be state-dependent, interface-based, and monotonic. The details of this specification are left to the paper. On a general level, an action sequence SIM-commutes (and subsequently scales) if its operations can be rearranged in any manner to produce the same outcome.

With this definition in mind, the authors contribute advice on designing commutative interfaces. The first is to decompose compound operations. POSIX APIs often combine contain several sub-operations which do not commute. If possible, these operations should be separated. The second is embrace specification non-determinism. This allows for less synchronization, and more concurrency. Third, many interfaces require strict ordering, which limits commutativity when only weak ordering may be necessary. Finally, releasing resources asynchronously means expensive synchronization can be forgone, and commutativity can once again be improved.

The final major contribution of the paper is the COMMUTER system. The system allows a developer to specify interfaces before the implementation process. COMMUTER can identify test cases using these interfaces that commute. As a result, the developer can make global interface changes to improve scalability before the implementation process.

4. Evaluation
The authors provide a great deal of evaluation. They show that their MTRACE tool identifies that out of 13,664 Linux test cases, only 9,389 are conflict-free. In many cases, shared reference counts are to blame. To improve this, their sv6 kernel was modified using the COMMUTER system. As a result, 13,528 of the same test cases now scale, a large improvement.
Additionally, tests are run to determine whether commutative operations actually scale. Using the sv6 system, both statbench and openbench shows that new commutative APIs are much more scalable than the traditional Linux versions. They also find that a mail server can achieve 7.5X scalability when transitioning from 1 to 8 sockets.

5. Confusion
Is the COMMUTER tool checking every single pair of interface APIs? If so, how is this computationally feasible?

1. Summary
This paper presents COMMUTER, a tool for designing scalable software interfaces using analysis of high level interface models before an actual implementation has been created. The authors used COMMUTER to develop a experimental operating system called sv6 which improves on 18 of POSIX's software interface calls to improve the scalability.
2. Problem
Without COMMUTER, the common way of determining and interface's scalability and improving it is to test it with a given workload on a varying number of cores and then use a tool like differential profiling to find bottlenecks and improve them before re-testing. The authors aim to find a more general approach, and one that can be done earlier in the development process before the actual implementation has taken place.
3. Contributions
The authors use the idea of SIM commutativity when evaluating interfaces with commuter, which is a way of checking commutativity for a specific system state with a specific set of arguments, rather than all general states. This makes the commutativity easier to evaluate for complex real world interfaces. The tool the authors design is called COMMUTER, a Python program which analyses a model of an interface before an actual implementation has been made. It then provides a set of conditions under which the interface commutes. It also includes tool which can generate test cases for a real world implementation based on the results from the symbolic model, with another tool called MTRACE able to run the test cases and identify memory access conflicts in real code.
4. Evaluation
The authors evaluate sv6, an experimental operating system with a more scalable in-memory filesystem and a new virtual memory system. It is compared to the POSIX implementations of the interfaces and it is shown that the sv6 implementations scale for 99% of of cases, which the POSIX interfaces are only conflict free in about 68% of them. Benchmark data is then also provided comparing the performance of the two.
5. Confusion
How much work is developing a model of an interface compared to creating an actual implementation? How much simplification is possible.

1. Summary
This paper considered scalable commutativity from an interface-based view instead of an implementation-based one, proposed a rule that interface operations can be implemented in a scalable way whenever they commute (are conflict-free), and designed a tool COMMUTOR to automatically analyze commutative conditions and test commutative operations.

2. Problem
Recent scalability evaluation is highly implementation based, which may expose bottlenecks for a certain workload and concurrency, but it does not address the underlying fundamental problems. This problem can be resolved by examining scalability from a higher level at software interfaces.

3. Contributions
There are two major contributions of this paper: a theoretical analysis of scalable commutativity rule, and a COMMUTOR tool that automate scalability condition analysis and testing.
In a shared-memory system, operations commute if one core does no writes to a cache line that is read/written by another core. Under this conflict-free scenario, interface operations commute in a scalable way.
A formal SIM commutativity is a state-dependent (S), interface-based (I), and monotonic (M) property that captures the idea that the order of operations does not matter if they are indistinguishable to future actions. And under the assumption of SIM commutativity, there exists a correct implementation that is conflict-free.
To exploit the rule, existing POSIX interfaces must decompose compound operations, allow nonderterminism and weak ordering, and release resources asynchronously. This involves modification to existing interface semantics.
COMMUTOR is a tool that automates commutativity rule reasoning. It has three components. ANALYZER will extract commutativity rules by examining all permutations of operations yield the same resulting state. TESTING will generate test cases on all possible paths given the commutativity rules. And MTRACE runs the testing and checks the result.

4. Evaluation
This paper looked for possible scalability opportunities for 18 POSIX APIs on file and virtual memory systems on Linux and sv6 using COMMUTOR. And it found that 68% and 99% cases are scalable for Linux and sv6 respectively. It then evaluates scalability performances on two microbenchmarks and a real application. The results for microbenchmarks show that modified interfaces scale very well compared to original implementations. And the evaluation on mail server application shows almost a linear scalability if using any file descriptor.

5. Confusion
In the scalable commutativity rule it assumes the specification CurlyS contains all correct orders. It may be a very large set of exponential size. When it comes to implementation of COMMUTOR, how do they generate the set CurlyS and how do they know which orders are correct and which are not?

Summary:

This paper sets out to design a scalable system with the help of a rule identified by the authors. The rule simply states that system interfaces that possess the property of being commutative are scalable. Based on this rule, the authors develop a system called Commuter. Commuter is a tool that the authors use to test the scalibility of system calls and ultimately practically prove the rule they identified.

Problem:

The problem that this paper ties to solve is system support for scalable computing. It's an interesting paper, in the sense that API calls aren't the first thing that comes to mind when I think of scalability issues.

Contributions:

The biggest contribution of this paper is the concept or rule that commutativity implies scalibility. This is based on the simple fact that operations whose output are independent of order do not need to communicate. The authors then go on to prove this rule theoretically and practically.

Commuter seems like an interesting tool that tests the scalibility of system calls through Python models and test case generation. The authors use this tool to demonstrate the predictive scalable nature of Linux and sv6. A tool that can predict how scalable an interface is could be useful in system design.

The authors were able to design a kernel that uses this rule in it's implementation. sv6 demonstrably (granted with the authors own tool & limited microbenchmarks) proves the commutativity scalable rule.

Another interesting aspect of the paper is the manner in which sections 3 and 7 are written. Section 3 talks about a few problematic API calls while section 7 explains how they constructed microbenchmarks that make use of these API calls.

Evaluation:

To evaluate and hence prove their rule, the authors implemented a scalable version of the xv6 kernel that they dubbed sv6. In addition to POSIX APIs that have the commutativity property, they implemented a file system and virtual memory system that did so as well. They chart the performance for two (three really) benchmarks and a mail server with an increasing number of cores (up to 80).

Their results show that sv6 is quite scalable but the microbenchmarks that they use only look at a small subset of APIs, specifically ones related to file descriptors and sockets. With that being said, their results do prove that their commutativity-scalibility rule has a lot of merit. In some ways, these tests could be described as worse case scenarios.

Confusion:
What does this paper mean to distributed systems?

1. Summary

The authors offer a reasonably simple and verifiable condition for an interface to be scalable, centered on the concept of commutativity. Furthermore from this condition, they have create a system to check the scalability of an interface by creating test cases from a model of the interface. They then use this framework to consider the scalability of a subset of the POSIX APIs.

2. Problem

One can wonder about the degree to which an interface, is scalable. This itself is a non-trivial question, and one with practical considerations in that scalable software should be built around scalable systems. And to this end, one approach to these questions is to ask what is a reasonable condition which implies the scalability of a system? And to what degree do real world systems satisfy these conditions?

3. Contributions

To model an API, they view execution as a series of actions, each action being either an invocation or a response. Which from the perspective of an API can be viewed as the call, and the return. Then a computation is a series of these actions, possibly in multiple threads, satisfying some set of rules. Then within this framework, they prove that commutativeity implies scalability. Furthermore, they give a constructive proof that shows for any particular history, how one might implement the API such that it is scalable with respect to the history. As would be reasonably expected though, the construction in the proof usually fails to yield effective implementations, and furthermore these constructions can be so history dependent as to be all but impossible to implement scalably in all cases.

Nonetheless, they manage to use the above rule to implement a framework for checking and testing the scalability of interfaces and their implementations. This framework works from a python model of the interface, and via symbolic execution, generates conditions on the commutativity of the various interfaces, from which test cases are generated. These test cases are can be run, while watching memory accesses, to verify or refute the scalability of various system calls. Here there model of scalability is that a set of operations scales if the memory accesses are conflict-free. That is to say, during the operation no two cores accesses a cache line accessed by another core.

4. Evaluation

To test their framework, they analyzed a subset of the POSIX APIs with it, finding significant scalability issues with Linux's implementation as compared with sv6 where almost all of the tested cases scale. This at least demonstrates that their framework can effectively analyze interfaces and give results with good granularity. What is not clear from this is if Linux actually scales significantly worse on any benchmarks. In particular, we certainly have the test cases generated by COMMUTER, but no demonstration that these are reflected in actual scaling.

Regardless, they do give us reason to believe that the commutativity law is an effective one. In particular, for sv6, the ran a few benchmarks with two slightly different APIs one commutative and one not. And here, the data convincingly shows that at least for programs that use these APIs heavily, we can achieve near perfect scalability with commutative APIs whereas the non-commutative APIs clearly fall short. These test are not perfect either, since even the mail server benchmark is rather specialized, and thus it is not clear in general how much benefit can be obtained from the API change and whether such a change would be worth the effort.

5. Confusion

To what extent to do real world systems with >20 cores show up, as this is where we see serious improvements in the benchmarks for sv6?

The scalable commutativity rule: designing scalable software for multicore processors
1. Summary
This paper introduced and proved the scalable commutativity rule - “whenever interface operations commute, they can be implemented in a way that scales.” In the paper, the idea is explained with formal definitions and proofs. Common methods for designing commutative interfaces are discussed. And a new tool COMMUTER is invented to evaluate the scalability of an implementation with automatic analysis and testing.

2. Problem
The major problem to solve in this paper is to understand the scalability of a piece of software. Differential profiling was used to identify scalability bottlenecks, but there are several problems with this method:

• Different workloads or higher core counts could introduce new bottlenecks.


• This method could only be used late in the development process. We hope to find out scalability bottlenecks at design level.

  3. Contributions
This paper presented the idea of scalable commutativity rule, declaring a set of operations scales if their implementations have conflict-free memory accesses, and communication could help avoid memory access collisions between order-irrelevant operations. Therefore, if such interface operations could commute, scalability could be achieved.

Besides, the paper also pointed out that to design scalable interfaces, there are several methods that can be adopted to improve commutativity:

• Decompose compound operations


• Embrace specification non-determinism


• Permit weak ordering


• Release resources asynchronously

Finally, the paper tried analyzing interfaces of a system using COMMUTER, a new tool to understand scalability. There are three major parts in COMMUTER: Analyzer, Testgen, and Mtrace.

 
  4. Evaluation
The paper evaluated the scalability of 18 POSIX calls using COMMUTER. The result has shown that Linux does not scale in many cases, and the paper uses the results to guide the implementation of a new research OS kernel called sv6.
 
  5. Confusion
I didn’t understand “Embrace specification non-determinism” in section 4.

The scalable commutativity rule: designing scalable software for multicore processors
1. Summary
This paper introduced and proved the scalable commutativity rule - “whenever interface operations commute, they can be implemented in a way that scales.” In the paper, the idea is explained with formal definitions and proofs. Common methods for designing commutative interfaces are discussed. And a new tool COMMUTER is invented to evaluate the scalability of an implementation with automatic analysis and testing.

2. Problem
The major problem to solve in this paper is to understand the scalability of a piece of software. Differential profiling was used to identify scalability bottlenecks, but there are several problems with this method:

• Different workloads or higher core counts could introduce new bottlenecks.


• This method could only be used late in the development process. We hope to find out scalability bottlenecks at design level.

  3. Contributions
This paper presented the idea of scalable commutativity rule, declaring a set of operations scales if their implementations have conflict-free memory accesses, and communication could help avoid memory access collisions between order-irrelevant operations. Therefore, if such interface operations could commute, scalability could be achieved.

Besides, the paper also pointed out that to design scalable interfaces, there are several methods that can be adopted to improve commutativity:

• Decompose compound operations


• Embrace specification non-determinism


• Permit weak ordering


• Release resources asynchronously

Finally, the paper tried analyzing interfaces of a system using COMMUTER, a new tool to understand scalability. There are three major parts in COMMUTER: Analyzer, Testgen, and Mtrace.

 
  4. Evaluation
The paper evaluated the scalability of 18 POSIX calls using COMMUTER. The result has shown that Linux does not scale in many cases, and the paper uses the results to guide the implementation of a new research OS kernel called sv6.
 
  5. Confusion
I didn’t understand “Embrace specification non-determinism” in section 4.

Summary:
This paper talks about the scalable commutativity rule for multicore scalability based on software interface. A software implementation of this rule based software is used to evaluate existing interfaces, and an sv6 implementation shows the practical use of this software.

Problem:
The scalability of a system can be limited by the software interfaces due to conflicted memory accesses in concurrent operations. If these conflicts are identified before interface implementation, it will be easier to design scalable solutions. Using existing approach of plotting performance for workloads identifies bottlenecks that are not necessarily fundamental to scalability.

Solution:
The tool(COMMUTER) developed to address the scalable interface design is based on scalable commutativity rule which states: If interface operations commute, scalability is ensured. The important points of this rule are:
-For commutativity to hold for a interface the order of actions(invocation or response) can be any of any order.
-SIM commutativity captures state dependence and thus can identify the states in which an interface can commute. It is monotonic meaning the subset P of any ordering Y also commutes in a history, H = X || Y.
-A correct implementation will generate responses allowed by specification(prefix-closed set of well-formed histories)
COMMUTER has three important components:
-ANALYZER that generates the condition under which the interface operations commute.
-TESTGEN that generates test cases for the conditions generated above and chooses the least set by identifying isomorphic group of test cases.
-MTRACE that checks if implementation is conflict-free for the test cases.

Evaluation:
The paper uses COMMUTER to evaluate Linux kernel on POSIX file system and virtual memory calls and identifies that in 30% cases scaling is not conflict-free. The paper then evaluates the use of this tool in designing the interface for sv6, for which more than 98% of interface implementation scales well based on generated test cases.

Concerns/Questions:
I did not completely understand the non-monotonic aspect of SI-commutativity discussed as example in paper.

Summary:
This article talks about achieving better scalability while the design of software interfaces. The emphasis is on commutativity where the order of actions does not matter. The authors have developed a tool called COMMUTER which has a test-driven approach. It applies commutativity rules to figure out which operations are scalably commutative. COMMUTER uses another tool called the ANALYZER which looks at path and conflict conditions required to make an existing interface commutative and scalable.

Problem:
Scalability issues should always be kept in mind while designing interfaces to cope up with the rapid increase of cores. If done too late many scalability bottlenecks surface and they are hard to take care of at that stage. This paper introduces a tool called COMMUTER which helps in measuring an interface's scalability.

Contributions:
1. The formal proof of SIM commutativity is given which states that whenever there is commutative interface operations, there is a way to implement them in a conflict free manner.
2. SIM (state-dependent, interface based, monotonic) commutativity does a few things to achieve scalability. First it decomposes bound operations. An example would be to separate program copy from new process generation while forking. Secondly, make specifications a bit non-deterministic. For example while opening a file, it is not always necessary to provide the lowest possible file descriptor. Thirdly, permit weak ordering. Fourthly, one hast to release resources asynchronously. For example, grouping all TLB shootdowns and doing it asynchronously scalability can be realized.
3. COMMUTER employs the services of many other sub-tools for the determination of commutativity and scalability. ANALYZER - It accepts an interface specification, written in a variant of python. It covers all the path conditions and outputs the commutativity conditions. TESTGEN - It generates test cases for the generated commutativity conditions by ANALYZER. MTRACE - IT runs all the generated test cases by TESTGEN and finds out how many of them fail thus determining scalability of that interface.

Evaluation:
Various POSIX file system and virtual memory calls have been modelled in COMMUTER. 18 POSIX APIs were compared to the sv6 APIs. sv6 is conflict-free for 13,528 of the 13,664 tests, while Linux is conflict-free for 9,389 tests. Many other benchmarks like statbench, openbench and a mail server have been used to prove that commutative interfaces scale very well.

Confusion:
I am not able to follow the formal proof of SIM commutativity. Also, how far have real world and very popular kernels like Linux gone ahead and done something like this.

Summary:
This paper talks about a new methodology for dealing with scalability issues during software interface design. The authors describe a rule for identifying scalability opportunities using just interface definition and behavior and provide an automated framework for identifying those scenarios and testing if an implementation properly scales in them.

Problem/Motivation:
The authors feel that dealing with scalability once implementation of an interface has been completed is too late to be effective as many scalability bottlenecks result from the design of the interface itself. They also feel that though there have been attempts to deal with scalability during software design, those attempts have been too specific (i.e. focused on a specific kind of system) and there has been no general methodology to follow for dealing with scaling during software interface design.

Contributions:
-The first major contribution of the paper is the development and formal proof SIM commutativity rule which states that whenever interface operations commute, they can be implemented in a way that scales (/conflict-free).
-SIM commutativity is state-dependent, interface-based(rather than implementation based) and monotonic. The state-dependence offers many more scenarios where the operations can commute than what a traditional(general) notion of commutativity would.
-The authors prove their formal statement of the above mentioned rule by generalized construction of a scalable implementation for a SIM commutative region of a invocation(action/response) history sequence.
-The second major contribution of the paper is the automated 'Commuter' framework for identifying commutative conditions ( analyzer module), generating testcases for checking if a particular implementation scales in those conditions(testgen module) and running those tests ( mtrace module).

Evaluation:
-The authors describe their redesign and implementation of a subset (18) of posix calls in a kernel based on xv6 ( called sv6) that allowed the calls to be more commutative (and hence scalable in theory). They also mention that they extensively used the 'Commuter' to guide their design and implementation (of some parts).
-They present the result of running a couple of micro-benchmarks (statbench and openbench), and a mailserver (with multiple communicating processes). Their evaluation compares the scalability of the original posix API vs. their redesigned API and they show that the new API out-scales the older in all scenarios.
-They also argue through examples that the improvement in performance cannot be obtained through implementation changes alone unless there is a fundamental change in the interface behavior to make it more commutative.

Confusion:
-I am not fully clear on how the testgen module isomorphically groups the possible assignments. Can you please talk about that?

Summary
While a lot of work has been towards scaling up operations and increasing throughput using concurrency and parallel processing, this recent work highlights the importance of commutativity (where execution order doesn’t matter) for making interfaces more scalable. The main concept in this paper is maximizing scalability by exploiting commutativity. The authors also developed a tool called COMMUTER which follows a systematic test-driven approach, applies commutativity rules to determine which operations when commutative can scale the operation.

Problem
Scalability is always seen as a characteristic of implementation. Evaluating scalability on multicores involves choosing workload, plotting performance curves, use different profiling tools etc and hence tedious. These detailed processes may in turn have issues that may be hard to find or are discovered too late into development. This papers aims at providing a way to measure an interface’s scalability implementation. Identifying scalability problems prior to implementation and during interface design is challenging. Such an approach is very useful is design and testing. COMMUTER attempts to evaluate some of these factors.

Contribution
The paper describes the commutativity rule – “Whenever several operations commute at the interface-level, implementations without conflicts in memory accesses can be achieved”. The idea here is to use commutativity as a measure of concurrency. If the order of operations is not important then we can always implement a lock-free/wait-free interface. When operations commute at the specification level, this rules tells that they should persist because scalable implementations exists or don’t otherwise. This fact is very useful in design of any software. Operations may be modified and redesigned such that they commute to ensure scalability.

A clear model uses actions which is a combination of invocations and responses. The system goes from one to state to other when a system call or invocation happens. This model helps to formally define the SIM commutativity rule. Being a paper from recent times, various POSIX issues are pointed out. Practical examples are important and allow the reader to better connect idea and usefulness of the model to their area of relevance.

Next comes the COMMUTER tool. The tool accepts high-level interface models and generates tests of operations that commute and hence could scale. It consists of 3 parts – Analyzer, Testgen and Mtrace. ANALYZER takes a symbolic model of an interface and computes precise conditions under which that interface’s operations commute. TESTGEN takes these conditions and generates concrete test cases of sets of operations that commute according to the interface model, and thus should have a conflict-free implementation according to the commutativity rule. MTRACE checks whether a particular implementation is conflict-free for each test case.

Evaluation
A detailed and exhaustive evaluation has been done, (another reason this is as a well written paper). After modelling several POSIX file system and virtual memory calls in COMMUTER, used both to evaluate Linux’s scalability. 18 POSIX APIs were compared to the sv6 APIs. C OMMUTER determines that sv6 is conflict-free for 13,528 of the 13,664 tests, while Linux is conflict-free for 9,389 tests. To further confirm the scalability on real hardware, evaluations were done with some microbenchmarks and mail server application. Using statbench, compare link/unlink with stat. Openbench is used to check the scalability of file open operations and file descriptor allocations. In both microbenchmarks, the commutative version scaled over their non-commutative counterparts. Mail server has lot of mix of operations, non-commutative operations cause the benchmark’s throughput to collapse but the commutative configuration scale reasonably.

Confusion
This paper although surely is nicely written but the concepts of commutativity and their relation to scalability appeared a little vague to me. I do not understand their correlation :(. I never read a paper in this topic. To name a few, I do not understand the clear distinction of interface, operation and implementation.

Summary:

The paper describes the development of a tool named COMMUTER which helps to identify if an interface scales well to multiple cores, provided the interface specification. COMMUTER uses an ANALYZER to analyze all the path conditions and the conflict conditions and outputs the commutative conditions required to make the interface commutative and hence scalable.

Problem:

Today, the number of cores is increasing rapidly and software has to scale correspondingly if the equivalent performance is to be acquired. There is no way of identifying if an interface is going to scale before implementing it. An early identification mechanism helps to reduce the amount of effort needed to scale an interface.

Contributions:

COMMUTER uses commutativity as a guarantee for scalability. To be precise, it uses SIM (state dependent, interface based, monotonicity) commutativity to guarantee scalability. The paper provides the following steps to achieve commutativity:
1. Decompose compound operations - An interface like fork which creates a process and copies the address space, should separate the two actions i.e., just create a process first which is a scalable operation. The idea here is fork is generally followed by exec and we can avoid the unnecessary non-scalable operations.
2. Embrace specification non-determinism - Many interfaces are more specific than required. Like the file open interface returns the lowest fd which make it difficult to scale. Relaxing the condition helps make it scalable.
3. Permit weak ordering - For interfaces like send and recv, by relaxing the ordering on send and recv we can make it more scalable.
4. Release resources asynchronously - Operations like remote TLB shootdown is not scalable since for every virtual processor switch one would have to wait. By making it asynchronous and grouping all TLB shootdowns scalable solutions can be obtained.

COMMUTER uses the following three sub-tools to determine if an interface design is going to be scalable:
1. ANALYZER - It takes in an interface specification, written in a symbolic variant of Python, covers all the path conditions and returns the commutativity conditions.
2. TESTGEN - It generates test cases for all the generated commutativity conditions. It covers all the paths possible to achieve a particular condition and all the kinds of access patterns on the shared data structure in the interface.
3. MTRACE - It runs the test cases using multiple cores and finds out how many of the tests fail thereby determining how scalable the interface is.

Evaluation:

The paper provides a very good view of commutativity and provides concrete examples of existing interfaces to help understand the problems and the solutions. The examples make it a lot easier to understand the concepts. Also, the performance evaluation clearly shows that commutativity helps achieve scalability.

Confusions:

The proof for SIM commutativity equals scalability is a bit confusing at certain points.
What is symbolic variant of Python? How exactly is the interface specification provided to the COMMUTER?

Summary
Often scalability of an operation is thought as a property of its implementation and not of its interface. But, this paper introduces a new rule that, if the interface operations are commutable, then there exists an implementation that allows scalability, thereby connecting scalability to interface commutativity. Operations are considered commutable if the results and system-state at end are independent of the order of the operations. A new tool named COMMUTER is introduced. The tool accepts a interface model and generate test cases with commutable operations that can be used to check the scalability of an implementation.
Problem
The authors claim that evaluating the scalability of software based on workload performance on varying number of cores has several drawbacks. It is difficult to identify the cause of the bottleneck; whether it is because of the implementation or is it because of the interface or hardware itself. And many a times the scalability is tested after the implementation leaving developers no time to redesign the interface or implementation. So it is better we identify the scalability of the interface before developing the implementation.
Contribution
Commutativity means the order of set of actions doesn't matter. The main contribution of the paper is SIM commutativity rule. This is the monotonic version of the commutativity rule and restricts the rule to particular system context, operations and arguments. However it guarantees that there exists a scalable implementation, given concurrent and commutative operations of interface.The paper also introduces the COMMUTER tool. ANALYZER component takes the interface model as input and outputs the precise conditions under which the interface operations commute. TESTGEN component takes these conditions and creates test cases consisting of commutable operations. These test cases can be used to check the scalability of the implementation. The third component MTRACE checks whether a particular implementation is conflict free for each test case.
Evaluation
The COMMUTER tool is applied to 18 POSIX calls and test cases were generated to check the scalability of linux. Linux scaled for 68% of the tests. The authors used these to guide the implementation of a new research kernel called sv6. They evaluated the performance using two benchmarks statbench and openbench. The benchmarks had two variants, one using the standard POSIX API's and the other that accomplishes the same task using modified commutative API's. commutative APIs scaled at almost an order of magnitude compared to a system that does not.

1. summary
This paper presents new approach to scalability that starts at a higher level : the software interface. At the core of their approach is this scalable commutativity rule: "whenever interface operations commute, they can be implemented in a way that scales". This rule lead to a new way to design scalable software: first analyze the interface's commutativity and then design an implementation that scales in commutative situations. To help in designing complex interfaces a tool called commuter is also introduced. It takes an interface model computes precise conditions under which sets of operations commute. To demonstrate the value of the rule and commuter, they explore the commutativity of POSIX and propose modifications to allow greater scalability.
2. Problem
The state of the art for evaluating the scalability of multicore software is to choose a workload, plot performance at varying numbers of cores, and use tools such as differential profiling to identify scalability bottlenecks.This has several drawbacks. Different workloads or higher core counts often exhibit new bottlenecks, making it unclear which bottlenecks are fundamental. This makes developers give up without realizing that a scalable solution is even possible. Also, this process happens so late in the development process that design-level solutions such as improved interfaces are impractical.
3. Contributions
The main contribution of the paper is the use of commutativity to reason about scalability. The scalable commutativity rule is presented, formally defined and proved. The rule relies on SIM (State-dependent, Interface-based, Monotonic) commutativity whose generality broadens the rule's applicability to complex software interfaces. The paper also describes a tool called COMMUTER for applying scalable commutativity rule to complex interfaces to aid developers. COMMUTER has three components. Firstly, Analyzer takes a symbolic model of interfaces and computes conditions under which interface's operations commute. Secondly, Testgen takes these condition and generate test cases. Third, mtrace checks whether a particular implementation is conflict free for each test case.
4. Evaluation
Performance evaluation is done using sv6, a POSIX level kernel using commutable APIs. Out of 12,664 test cases generated by commuter, linux scales only for 68% while sv6 for 99% of test cases. In both statbench(fstat vs fstatx) and openbench(open with/without O_ANYFD) microbenchmarks, commutative version scale linearly while the non-commutative version restricts scalability. Also in Application benchmark(regular API vs commutative API) running a mail server, the results are similar. Non-commutative operations cause benchmark's throughput to collapse at a small number of cores while the configuration using commutative API achieves 7.5x scalability.
5. Confusion
Didnt understand the monotonic requirement of the commutativity rule.

Summary: This paper introduces a novel way of examining the scalability of a software and how to design scalable interface. They achieved this by realizing commutativity will bring scalability and they developed a tool that tests the commutativity of interface.

Problem: Scalability is important to modern multicore CPU architecture. However examining the scalability of a software or interfaces is hard because (1) it depends on the implementation; (2) it depends on the tests that have been used. Moreover, such evaluations do not generally provide any advice to improve the scalability.

Contribution:
1. Draw connection between the scalability of a software interface and its commutativity. They claim that if two system calls are commutative, then they are scalable. Here commutative means swapping the order of these two calls cannot be observed using system calls. If two system calls are commutative, then there must be some scalable implementation for them. Using this principle, we can examine the scalability of an interface without requiring its implementation. This also brings opportunities to automate such examinations.

2. Develop a tool called COMMUTER that can quantify the scalability of an interface and generate tests for commutativities. By employing this tool, one can discover what is the bottleneck of the interface, which can be used as guidance to a better more scalable design.

3. Identified many places in linux interface that restricted the scalability of Linux. These include allowing for non-deterministic behaviors, weak ordering, and asynchronized behaviors. Currently few program rely on these assumptions, but these assumption will greatly harm the scalability of the system. By discarding these assumptions, certain system calls can be implemented more scalable.

Evaluation: They ran their COMMUTER on linux and discovered that the linux interfaces scale for 68% of the tests generated by COMMUTER. In the meanwhile, they proposed a new interface includes 18 POSIX compatible functions. In comparison, the new interfaces were 99% scalable.

Confusions: They mentioned they used symbolic execution to reason the commutativity of a given interface. This part seems black magic to me, and I do not understand it.

Summary :
The paper talks about how software could be made scalable on multicore processors using the scalable commutativity rule. They prove the usage of the technique by developing a tool called Commuter that generates test cases for operations that commute and thereby can test the implementation and figure out scalability bottlenecks. An experimental kernel sv6 has been developed by using the results of commuter to tackle scalability and it has been seen to scale for 99% of the test cases as compared to Linux that scales for 68% of them.

Problem :
The problem that is being dealt with is to be able to scale multicore software by being able to identify the scalability bottlenecks from the interface specifications and thereby modify the implementation such that it scales.

Contributions :
1. The main rule that they propose is the scalable commutativity rule i.e if the interface operations commute, they could be implemented in a way that scales.
2. The rule lead to a process wherein the commutativity of the interface is analyzed and then an implementation could be designed that scales in those commutative situations.The SIM commutativity rule has been proved using formal proofs.
3. They propose some basic principles to design commutative interfaces such as
fine grained operations that would improve the commutativity between operations
Allow weak ordering unless strict ordering of operations is necessary
4. The Commuter tool has been developed to analyze the interfaces. The Analyzer considers the interface specifications and computes the conditions under which they would commute. Test cases are generated by TestGen taking these conditions into account and Mtrace checks for conflict freedom in the implementation for each test case.
5. Commuter has been used to test POSIX file system and virtual memory calls and identify the opportunities for scalability. The common problems identified were shared reference counts, coarse grained locks and directory level locks.
6. Some patterns identified for scalable implementations were datastructures such as radix arrays/hashtables that improve commutativity, per core resource allocation, double checked locking, avoid read operations that conflict with other concurrent operations unless required.

Evaluation :
The paper gives an extensive performance evaluation of both the Commuter tool that has been used to develop the sv6 kernel and also the scalability of the resulting implementation. The commuter tool generated 13664 test cases and the conflict freedom matrices for linux and sv6 show that sv6 could scale for 13528 cases as compared to 9389 in Linux. Also, the scalability has been evaluated using statbench, openbench and a mail server and the comparison of performance throughput with and without the changes for scalable implementation are shown. An increase in scalability by 27% in openbench and 7.5 times scalability for mail server using commutative API’s is observed.

Confusion :
Formal proof for SIM commutativity is not clear.
Have these techniques for scalable implementations been used in any of the mainstream OS implementations?

Summary:
The authors propose an interface-driven scalability approach such that whenever interface operations commute, they can be implemented in a way that scales. An automated test generation tool is implemented to verify scalability implementation of various interface operations.

Problem:
Current approach of workload-driven scalable development focuses on developer effort but has few disadvantages such as bottlenecks exposed by newer workloads or with the increase in number of cores. The authors identify that real bottlenecks are in the interface design while the workload driven approach focus on implementation. Hence to improve scalability, the authors stress on improving interface design.

Contributions:
1. The scalability commutativity rule and its formalization. SIM commutativity is state-dependent, interface and monotonic. It is sensitive to operations, arguments and state of operations.

2. Commuter: An automated scalability testing tool to verify if an iimplementation is conflict-free.
Three components:
- Analyzer models a particular implementation by generating all possible codepaths.
- TestGen generates testcases and varying inputs for codepaths generated by Analyzer.
- Mtrace module identifies conflicts (variables/operations) in the implementation.

3. Implementing a scalable POSIX-level kernel named sv6.

Evaluation:
The authors test the scalability implementation of POSIX system calls in Linux and sv6 through testcases generated by Commuter.
Out of 13,664 testcases generated, Linux scales for 68% of the testcases while sv6 scales for 99% of the testcases.
Scalability and Performance evaluations were done using two microbechmarks(statbench and openbench) and an application level benchmark to study the cost of non-commutativity and examine the scalability of conflict-free implementation of commutative operations. Commuatative versions of fstat and open scale linearly with increase in core count while their non-commutative counterparts limit scalability around 20 cores. In mail server, commutative APIs scale 7.5 times from 10 to 80 cores while non-commutatiive versions limit it to small number of cores.

Confusions:
I could not quite understand the importance of monotonicity for SIM commutativity rule. Also, How can this rule be applied to a distibuted environment setting?

Summary:

In this paper, the authors describe about designing scalable interfaces for multicore processors using a new approach called the scalable commutativity rule and discuss its proof. This rule is used to evaluate the scalability of the interfaces designed at the interface level. They also discuss about the Commuter tool which takes the interface model which generate tests using which the scalability of the implementation is evaluated. The authors also designed a new kernel called the sv6 using this method and compare its performance with Linux.

The primary problem that the authors are trying to address is how to evaluate the scalability of the software that runs on multicore processors. The prior methods involve designing a workload and evaluating it in various scenarios to identify the different bottlenecks. The authors propose an approach which tries to evaluate the scalability by taking the interface design into account.

The primary contribution is the idea to use the interface design to evaluate the scalability. The authors posit that a set of operations is scalable if their implementations are conflict free. They put forward the SIM commutativity which is implementation independent. They describe a new method of implementation where the interface’s commutativity is evaluated and implementation is done after that such that it scales to the commutative situations. The authors prescribe a test driven approach using the commuter tool to do implementations of interfaces. The commuter consists of three components. The ANALYZER takes as input a model of behavior of interface and outputs commutativity conditions to the TEST GEN which converges the test cases to concrete test cases. Since it does not know the exact input that can cause conflict, TEST GEN uses conflict coverage to capture different conflict conditions as well as path conditions. The MTRACE runs the test case on real implementation a real implementation and verifies whether it is conflict free.

The authors built a prototype of the COMMUTER and used it to implement POSIX file system and virtual memory calls in sv6 and compared it with Linux. The evaluation of the Linux file systems show that the commutativity rule is contradicted due to problems like access conflicts in shared reference counts and coarse grained locks. By using this knowledge the new file system (ScaleFS) and virtual memory system (RadixVM) were built. The overall scalability is 68% for Linux and 99% for sv6 on the tests generated by the COMMUTER. The sv6 also performs better in the stat, open benchmarks with better scalability as the number of cores increase. Also the performance of commutative APIs for a mail server designed scaled 7.5x better than the normal APIs.

It is said that TEST GEN partitions values into isomorphism groups to bound the enumerated test cases. How these groups are formed and how are values assigned to specific groups? I could not understand the reasoning given to disable hardware prefetcher when the benchmarks were run.

Summary:
This paper puts forward the idea that scalabale software can be conceived at an interface level if the operations are commutable - i.e. can take different orderings. The authors design a commuter tool which will help generate test cases to analyze scalability in terms of commutability.

Problem:
Traditional scalability evaluation works by choosing a workload and then profiling the application by varying the underlying system. The authors argue that scalability in terms of testing using workloads is inefficient since it is done too late to make significant modification and changes to the design. The problem being addressed here is testing for software scalability at a higher level without need of specific benchmarks or workloads.

Contribution:
The key idea proposed in the paper is to think at scalability of software in the perspective of formal semantics and orderings rather than implementation and testing. Scalability is reasoned at a higher interface level using formal semantics of the State (S) Interface (I) and Monotonic(M) SIM-commutativity rule. The authors then propose a set of guidelines to design SIM-commutative software which could guarantee conflict free access. A formal proof underlying this concept is presented in the paper.
Another key contribution is the commuter tool designed to automate their findings. The tool takes a symbolic representation and generates test cases to check for scalability and checks if the operations associated with the interface are re-orderable or not.
The authors also implement a scalable fast file system over sv6 mainly for testing and evaluation purposes.

Evaluation:
The implementation of COMMUTER was tested on Linux POSIX API and on a scalable file system over sv6. With the ~13K tests cases, only 68% tests are conflict free and scalable with Linux in comparison to about 99% of them are scalable on xv6 when designed using the SIM-commutavity guidelines using scalable data structures.

Confusions:
Does arguing for scalability in terms of interfaces guarantees to lead to scalable software in all cases? Are there any other factors which should influence the design decision as well?

Summary:
This paper puts forward the idea that scalabale software can be conceived at an interface level if the operations are commutable - i.e. can take different orderings. The authors design a commuter tool which will help generate test cases to analyze scalability in terms of commutability.

Problem:
Traditional scalability evaluation works by choosing a workload and then profiling the application by varying the underlying system. The authors argue that scalability in terms of testing using workloads is inefficient since it is done too late to make significant modification and changes to the design. The problem being addressed here is testing for software scalability at a higher level without need of specific benchmarks or workloads.

Contribution:
The key idea proposed in the paper is to think at scalability of software in the perspective of formal semantics and orderings rather than implementation and testing. Scalability is reasoned at a higher interface level using formal semantics of the State (S) Interface (I) and Monotonic(M) SIM-commutativity rule. The authors then propose a set of guidelines to design SIM-commutative software which could guarantee conflict free access. A formal proof underlying this concept is presented in the paper.
Another key contribution is the commuter tool designed to automate their findings. The tool takes a symbolic representation and generates test cases to check for scalability and checks if the operations associated with the interface are re-orderable or not.
The authors also implement a scalable fast file system over sv6 mainly for testing and evaluation purposes.

Evaluation:
The implementation of COMMUTER was tested on Linux POSIX API and on a scalable file system over sv6. With the ~13K tests cases, only 68% tests are conflict free and scalable with Linux in comparison to about 99% of them are scalable on xv6 when designed using the SIM-commutavity guidelines using scalable data structures.

Confusions:
Does arguing for scalability in terms of interfaces guarantees to lead to scalable software in all cases? Are there any other factors which should influence the design decision as well?

1. Summary
The authors present a design for making scalable systems. The authors use the concept of commutativity used at a higher level such as the software interface. Commutativity in essence means that if two operations commute, their results are independent of order thus making communication between them unnecessary.

2. Problem
Scalability is generally evaluated by choosing a workload, measuring the performance by varying the number of cores and then profiling to identify the bottlenecks. This ideology helps to identify only the bottlenecks whereas fixing them in the implementation level is an entirely different issue. If we assume scalability is an implementation feature, in a shared memory multicore processor with a cache MESI-cache coherence protocol, reading ad writing to the same line inhibits scalability as the hardware serialises the accesses. Thus, the authors find a need to move the scalability decisions to the interface level by using commutativity to identify the independent regions.

3. Contributions
a) The authors present and prove the commutativity rule for scalable multiprocessors that is bound to work. SIM commutativity is state-dependent, interface-based and monotonic when compared to the usual definition of commutativity.
b) The authors analyse interface by a commuter (Analyzer + TestGen + MTrace):
i) Analyzer automates and analyses the interface and outputs commutativity conditions that are expressions that say when operations can commute.
ii) TestGen generates actual C test cases for the commutable operations for conflict-free operation.
iii) MTrace runs the test cases generated by TestGen and checks if the implementation is conflict-free for every test.

4. Evaluation
The authors evaluate the scalability on the Linux OS and develop 13664 test cases. Out of them, 4275 test cases were not conflict-free. The common source of these conflicts is the access of reference counts in operations such as file name lookup and so. To remedy this, the authors develop a scalable file system in their system called sv6 based on xv6 and reduce the conflict regions to 136. The authors also show that their file system scales and provides higher throughput than regular APIs in a higher core system.

5. Confusion
If the system becomes scalable, shouldn’t the throughput increase with the number of cores increasing? Why do they drop down (although not considerably) in the mail server throughput?

Summary:
Paper is about new approach to build scalable systems. It defines a new Scalable commutativity rule which tells whenever operations commute they can be implemented in a way that it scales. Author talks about Commuter tool which is used to find scalable opportunities and a new sv6 OS in which they have implemented almost all interfaces as scalable.

Problem:
Traditionally, in multi-core system, iterative version to build scalable system was used. For a workload, plot of scalability is made, bottleneck were identified, they were fixed and same process was repeated. However, this approach has many problems. Bottleneck were identified at later stage and design and implementation needs to be changed to fix the problem. New workload or adding more cores may introduce new scalable problems which are not considered during design stage. Interface design itself may have bottleneck and lead to non-scalability.

Contribution:
I think biggest contribution of paper is scalable rule itself. Authors formalize the rule and also provides the proof of its correctness. Author states that commutativity is sensitive to operations, arguments and state. If the operations commute, results are independent of order and there will be no conflicts. In such scenario, there exists scalable implementation. Another contribution of paper is automated scalable testing tool called 'Commuter' which helps finding commutative opportunities in an interface. Analyzer a component of Commuter takes a symbolic model of an interface and generates list of commutative conditions. TestGen takes these conditions and generates test cases. Finally, Mtrace uses test cases generated on implementation to see whether a particular implementation is conflict free. I consider sv6 also to be contribution of this paper as it provides a modified kernel with scalable POSIX like kernel.

Evaluation:
Evaluation was done to check whether rules helps in building scalable system. Commuter tool was used to find scalable opportunities in Linux and was applied to 18 POSIX calls. 68% of test cases were conflict free on Linux and hence Linux was 68% scalable. However, in sv6 99% of test cases were conflict free.

Confusion:
I didn't understand monotonic part of SIM commutativity.

Summary
The paper primarily talks about developing scalable software by banking on interfaces that commute, i.e. the interface doesn’t define order of execution. The authors define a form of commutativity that is state dependent, monolithic and interface based, which allows defining operations implementation of which can guarantee conflict free execution. The authors also describe a tool, COMMUTER that enables developers to apply the rule of commutativity.

Problem
The existing approach to achieving scalability was to implement the software and use profiling tools to identify bottlenecks to scalability. The authors mention an alternative approach to development that enables developers to reason about scalability before implementation and testing on hardware.

Contributions
The authors define a new form of commutativity, SIM commutativity, which enables reasoning about how having commutative interfaces can lead to scalable software, which results from having a conflict free memory access. Another approach to this was to look for cache line conflicts, which serialized accesses through cache coherence protocols. The authors use the SIM commutativity rule to identify interfaces, APIs, in POSIX that are not scalable because they don’t satisfy the rule, and they discuss possible modifications that can make those interfaces scalable. There is an implementation model presented, which provides developers an approach to design interfaces. Finally the authors discuss the COMMUTER tool in parts- 1) ANALYSER, which takes in symbolic model of interfaces and provides conditions for commutativity; 2) TESTGEN- takes these conditions and generates test cases of operations according and 3) MTRACE- tests if the implementation is conflict free for all the test cases generated. The authors also developed a file and virtual memory system, ScaleFS and RadixVM, for sv6 kernel, which they use to evaluate their proposed approach.

Evaluation
Using experiments with benchmarks run on 80-core, Intel E7-8870 with 256GB RAM, the authors show that the proposed approach for, 18 POSIX calls, scales for 99% of the tests generated by TESTGEN as compared to linux which scales only for 68% of the tests.

Confusions
Can we discuss the proof of the commutativity rule?

Summary: They propose a sufficient condition for linear scalability: whenever interface operations commute, they can be implemented in a way that scales. Based on this rule, they design a tool called Commuter, which take the interfaces design as inputs and generate test cases that operations can scale. It can also check the scalability of an implementation. The experiments on POSIX calls and the guidance of implementation of OS kernel sv6 shows many scalability problems in Linux and better scalability of sv6.

Problem: previous evaluation of scalable software is choosing workload and plot the performance varying w.r.t. #cores. This design evaluation pattern has two problems: 1. It relies on specific workloads, and different workloads may show different problems. 2. It is too late if some problems are found as the evaluation is after development. Therefore, we may need a general rule to get the scalability given the high-level interface.

Contribution:
1. They show a general rule for scalability: whenever interface operations commute, they can be implemented in a way that scales (conflict-free).

2. Based on this rule, they propose SIM (State-dependent, Interface-based and Monotone) commutativity, which generalize the rule to complex software with multiple states with dependency.

3. Based on the SIM commutativity rule, they design Commuter, which contains Analyzer to calculate conditions that opetations commute; Testgen to represent these conditions to more interpretable test cases; and Mtrace to test if a specific implementation is scalable (conflict-free).

Evaluation:
The author built a implementation of Commuter and their own kernel sv6 which has scalable implementation of POSIX APIs in Linux. They compare the scalability of sv6 vs Linux on two microbenchmarks and a mail client application benchmark. On the first two benchmark, the linear scalability is achieved by sv6 and Linux get limited at around 20 cores. In application case sv6 is 7.5x better scalalbility than Linux when between 10 and 80 cores.

Confusion: The number of states can scale exponentially with #interfaces and #arguments, is it correct?

Summary:
This paper gives the definition of scalable commutativity rule and introduces the COMMUTER tool to apply the rule.

Problem:
The current scalability evaluation happens after the implementation stage. It also depends on the workload and the hardware, making the bottleneck difficult to locate. The author proposes a new approach to scalability that starts at the interface level. It can highlight inherent scalability problems and contribute to the implementation.

Contribution:
(1) The paper defines the SIM commutativity and the scalable commutativity rule. SIM commutativity is state-dependent, interface-based and monotonic. If SIM commutativity holds, there exist an implementation that is conflict-free for the state, the arguments and the concurrent operations.
(2) The paper illustrates the interface-level reasoning enabled by the rule via the POSIX example. Compound operations should be decomposed, such as fork and stat. Specification non-determinism can be used such as FD allocating. The system should also weak ordering and release resources asynchronously.
(3) The paper introduces the COMMUTER tool, which is composed of three parts: ANALYZER, TESTGEN and MTRACE. The ANALYZER computes precise commutative conditions. TESTGEN generates concrete test cases. MTRACE checks the implementation on each test case.

Evaluation:
The author implements ScaleFS and RadixVM for sv6 and run experiments on an 80-core machine. The paper gives data about benchmark throughput with varying core counts and shows that commutative APIs achieve 7.5X scalability from 1 socket to 8 sockets.

Confusion:
Does the complexity of ANALYZER and TESTGEN matter? I think the time to run ANALYZER and TESTGEN is exponential to the number of interfaces. Because ANALYZER has to consider all permutations and TESTGEN has to consider all the coverage. If the number of interfaces is very large, the tool is not very practical?

Summary
This paper introduces a rule by which commutativity can be used to judge the scalability of the system. The rule states that whenever the interface operations commute, they can be implemented in a way that scales. Using this rule, a tool called COMMUTER was developed which could evaluate the scalability of the implementation by generating test cases for operations from high level interface models.

Problem
The current way of determining scalability was to run a workload on a system at varying number of cores and then measuring the performance. This method had a disadvantage that it doesn’t take into account that bottlenecks can be also caused by the operations and not just because of the hardware. Also the possibility of identifying scaling using the profiling tools can only be done by running the code on the actual hardware. The authors of this paper devised a way in which scalability can be determined even before the hardware is available.

Contribution
The authors define a rule called the scalable commutativity rule which states that if several operations to an interface cannot be distinguished by their execution order (commute) then they can be scaled — as their memory accesses are conflict free. These commutative operations do not require any form of communication between them hence this results in conflict free implementation. This rule provides a new way of designing scalable software by analyzing the interface’s commutativity. This is handled by the tool called COMMUTER, it computes conditions in which the operations commute. This is done by generating test cases and running them against OS POSIX calls. They also introduce a new form of commutativity called SIM commutativity — state dependent, interface based and monotonic.

Evaluation
The authors developed a prototype of COMMUTER whose test cases were run with the Linux kernel and their self developed OS sv6. It was determined that for 18 POSIX calls, Linux scales 68% of COMMUTER’s test cases however sv6 scaled for 99% of the test cases as these calls were improved to incorporate scalability.

Confusions
I was not able to completely understand the proof of the scalable commutativity rule.

Summary:
In this paper the authors present a new approach to identify and address scalability issues, by observing interface operations. Their argument is that if commutativity exists between operations then there is an implementation possible which has conflict free memory accesses and hence is scalable. The authors present the formal reasoning behind the commutativity rule and demonstrate a new tool COMMUTER that uses this principle to improve software scalability.

Problem:
The standard procedure for evaluating the scalability of software is to measure the performance on a given workload by varying the number of cores. The number of cores and the choice of the workload causes new bottlenecks which makes it hard to identify which are inherent in the software. Also this procedure happens late in the development process by which point it may be unfeasible to make design changes. Interface commutativity can be clearly reasoned out and can help identify the inherent problems to guide towards a conflict free implementation.

Contributions:
The core contribution in the paper was the formal reasoning to prove the scalable commutativity rule to make the claim that commutativity leads to improved scalability. The constraints of SIM commutativity were defined so it could be applied to the state dependent interfaces common to most software. SIM commutativity allows the actions of threads to be interleaved while maintaining the order within each thread but ensures that a prefix of actions are commutative as well so as not to rely on later operations in case the execution diverges from them. The authors also reason about some general rules in designing commutative interfaces by giving examples of POSIX APIs. Compounding return values, deterministic design and strict ordering between interface operations are some of the hindrances to commutativity that the authors identify with existing interfaces. The authors also develop the tool COMMUTER which automates the process of reasoning about commutative conditions and testing those cases to create a conflict free implementation for the interface.

Evaluations:
To test the scalability of the implementations identified by COMMUTER the authors built their own kernel sv6 which has the scalable implementations of the standard POSIX APIs found in Linux. The authors compare the scalability of the commutative interfaces against their standard counterparts using two microbenchmarks and a mail client application benchmark. The microbenchmarks statbench and openbench scaled linearly for the commutative interfaces while the standard interfaces was limited at around 20 cores. Similarly the commutative interfaces performed better on the application benchmark with a 7.5x improved scalability for around 10 to 80 cores.

Confusions:
Can you please explain the monotonic constraint of SIM in class.

1. Summary
The paper describes the scalable commutativity rule which states that whenever operations in an interface commute they can be implemented in a way that scales. The authors provide the formal proof of the rule and also practical application of it in the sv6 research kernel.

2. Problem
Measuring scalability of software over many cores is a challenge. The existing process suffers from the drawbacks of difficult and late identification of performance bottlenecks. This paper aims to solve this problem.

3. Contributions
This paper presents the scalable commutativity rule. The authors tie the scalability of software interfaces across many cores to the commutativity of its operations. They claim that if the operations of an interface commute, they can be implemented in a way that scales across many cores without performance degradation. The paper formulates SIM commutativity to use in software systems and then provides a formal proof of the scalable commutativity rule using SIM commutativity. The other major contribution of this work is the analysis of the linux kernel to find scaling opportunities and the implementation of the sv6 research kernel that is built using the scalable commutativity rule. The COMMUTER tool is developed to analyze interfaces to look for opportunities where commutativity can be exploited to get a more scalable implementation.

4. Evaluation
The authors test the sv6 implementation which is based on their proposed rule with the help of two microbenchmarks and one practical application. They provide the performance of unmodified linux kernel on a single core as a comparison point. The experiments show that sv6 scales lineraly and does not suffer from performance degradation as the number of cores increases.

5. Confusion
I did not quite understand the monotonicity requirement that the paper talks about.

Summary
In this paper, the authors introduce the idea of using commutativity as an interface design requirement to improve scalability. The authors demonstrates opportunities out there for more scalable interfaces and presents definitions and proofs of the scalable commutativity rule. The authors also describe a tool (COMMUTER) for automatically developing test cases and testing if a particular implementation commutes.

Problems
Traditionally, to evaluate the scalability of an application the developers choose a workload, test performance by varying number of cores and then profile it. However this approach is not exhaustive and different workloads or higher core counts often exhibit new bottlenecks. It’s unclear which bottlenecks are fundamental and moreover it happens too late for design level solutions. The real bottleneck may be in the interface design. This paper presents a new approach to scalability that starts at a higher level: the s/w interface.

Contributions
The core of the approach is the scalable commutativity rule which states that in any situation where several operations commute, i.e. there is no way to distinguish their execution order using the interface - they have an implementation whose memory accesses are conflict-free during those operations. SIM (State-dependent, Interface-based, monotonic) commutativity is independent of any specific implementation and allows developers to apply the rule to complex, stateful interfaces. The authors present formalization of the rule and proof of its correctness in paper. The authors presents a systematic, test-driven approach to applying commutativity rule to real implementations embodied in a tool named COMMUTER (components: ANALYZER, TESTGEN and MTRACE). This tool helps in addressing challenges developers face in complex interfaces where it’s difficult to spot and reason about all commutative cases given the rule.

Evaluation
The authors demonstrates the value of the commutativity rule and COMMUTER tool by applying commuter to a simplified model of 18 POSIX file system and virtual memory system calls. The authors used commuter to test the scalability for Linux OS kernel called sv6. Linux scales for 68% of the 13,664 tests generated by COMMUTER for these calls, and commuter finds many problems that have been observed to limit application scalability. sv6 scales for 99% of the tests. They also demonstrate their implementation on two benchmarks: Statbench and Openbench.

Confusions
Is this rule and tool helps in distributed environment? I was wondering if the tool components can be used elsewhere independently.

1. Summary
The paper proposes a method to implement scalable software by analyzing the state-dependent commutative property of its software interface. Scalability of the software here is implied by the conflict-free memory accesses of the implementation.
2. Problem
For a given software implementation, it is difficult to analyze and attribute the scalability issues of the software to the fundamental design decisions of the software design. Even if the issues are identified, it happens late in the development process, requiring a redesign of the implementation or the software interface. What lacks here is a formal way of analyzing the scalability of the software interface and a method to verify the scalability of implementation.
3. Contributions
The authors provide a formal way of expressing SIM commutativity. The idea is that the commutativity depends on the state of the system, the arguments and the concurrent operations. In addition, it is interface dependent and should be monotonic. Given that an operation is SIM-commutative, it is proved that a conflict free implementation exists for such operation.
This analysis of SIM commutativity can be done on the software interface, without detailed knowledge of the implementation. With this idea, the authors provide a tool that automatically generates test-cases that can be used to verify if a particular implementation is conflict-free. It involves three stages: Analyzer, to generate all the code paths of a model implementation, TestGen, to generate test-cases that cover the code paths for different inputs, MTrace, to run the test-cases and identify the conflicting operations or variables in an implementation.
4. Evaluation
The authors implemented POSIX APIs, a new file system ScaleFS and virtual memory system, RadixVM on a new sv6 kernel. The primary bottlenecks in the Linux FS implementation they identified were shared reference counts and coarse-grained locks. They implemented scalable implementations for these contended resources, for example by using data structures that satisfy the commutativity rule. With small microbenchmarks and an application, they noticed a scalable performance on an 80-core system. It would have been more interesting to see how the commutative interfaces interacted with non-commutative ones.
5. Confusion
How is the model implementation developed? Is it just a functional model without any data structures or system state? If it is without any system state, how can we show that it is SIM commutative?

Summary
The paper presents a method for detecting opportunities for scalability at the software interface level. The authors develop a tool called Commuter and talk about an operating system, sv6, that was developed with the help of this tool to maximize scalability.

Problem
A lot of research goes into making things scale well, and most look at benchmarks of implementations of systems when more cores are added. The authors argue that this is too late in the design process, if an entire system has already been built, it is too difficult to go back and introduce parallelism where there was not previously. Instead it would be much nicer if the parallelism of a system could be established during the design process where it is more easily modified.

Contributions
Primarily the paper contributes a method by which software’s parallelism or scalability can be reasoned about at the interface level, during its design. The authors wrote a program called COMMUTER which is capable of analyzing software interfaces and notifies developers of sections of code that are non-commutative. The authors provide some hints at simple optimizations that will allow for more scalability like reducing requirements for determinism in programs.

Evaluation
The paper includes results for runs of COMMUTER on the Linux kernel and the authors’ own sv6 operating system. These results claim that Linux is scalable for only 68% of the tests generated by COMMUTER whereas sv6 scales in 99% of the tests. These results do not give a clear indication, however, of how the operating systems will scale for average workloads. To this end, the authors include results of some actual benchmarks that corroborate COMMUTER’s results.

Confusion
Because COMMUTER is not implementation or benchmark specific, the tests may not be characteristic of how scalable an actual system may be for actual work loads. Is there some way to give COMMUTER hints as to what could be considered reasonable work to make the tests more in line with an actual implementation?

1. Summary
In the paper, "The scalable commutativity rule: Designing scalable software for multicore processors", the authors propose scalable commutativity rule to directly connect scalability to interface commutativity. They enlist the subtle problems of commutative interface design, particularly with respect to POSIX API calls. They develop a tool called COMMUTER that accepts high-level interface models and generates tests of operations that commute and hence could scale. Using this tool, they evaluate the scalability of 18 POSIX calls in Linux kernel and use the results to implement a new research OS, sv6.

2. Problem
Since scalability is considered as an implementation property i.e. scalability depends on hardware, the process of evaluating and reasoning about scalability of multicore software happens so late in the development process, usually after the implementation on the actual hardware. Can scalability options be identified before the implementation exists?

3. Contributions
SIM commutativity:
- The simple commutativity rule suggests to reason scalability at a higher level, interfaces. By constraining the commutativity rule to that of SIM Commutativity, one can reason about the scalability of interfaces(in practical, complex software) that commute in the context of specific system state, specific operations on arguments and specific concurrent operations.
- "Whenever interface operations commute, they can be implemented in a way that scales". In other words, whenever several operations commute at the interface-level (i.e., there's no way to distinguish their execution order using the interface), those operations are scalable.

Model:
- Model the system execution as a set of actions where an action is paired invocation(system calls with arguments) followed by a response(corresponding result of the system call)

COMMUTER:
- Analyzer takes a symbolic model of an interface and computes precise conditions under which that interface’s operations commute
- TESTGEN takes these conditions and generates concrete test cases of sets of operations that commute according to the interface model, and thus should have a conflict-free implementation according to the commutativity rule
- MTRACE checks whether a particular implementation is conflict-free for each test case

4. Evaluation
The authors have carried out an exhaustive evaluation by modelling several POSIX file system and virtual memory calls in COMMUTER and using this to evaluate Linux's scalability and to develop a scalable file (ScaleFS) and virtual memory(Radix VM) system for the sv6 research kernel. On running COMMUTER with Linux kernel, they found that only 68% of Linux was scalable whereas the sv6 was found to scale for 99% of the test cases. They also demonstrate their implementation on two benchmarks: Statbench: link/unlink versus stat, Openbench: file descriptor allocation and Mail server application.

5. Confusion
- Can you explain if the authors claim of this rule being applicable to the general distributed systems and not just multicore systems is valid?

Summary :
This paper presents a new rule, the scalable commutativity, to detect bottlenecks in applications and changing interfaces to make them more scalable. Based on this, a tool called Commuter has been developed and a linux-like OS, sv6 has been implemented to test out the improvements in performance.

Problem :
The authors state that it is very difficult to identify bottlenecks in running applications on multi-core processors. Especially with conflicting cache lines, the performance can be 25x worse even for simple system calls. And this only worsens with the increase in the number of processors. In this paper, the focus is on associating bottlenecks with interface design and trying to find scalability opportunities by changing interfaces.

Contributions :
1. The scalable commutativity rule, with the formalization. Apart from the other previous work in this line, an important contribution of this paper was to show how state dependent the commute is.
2. The tool, commuter, which is an automatic scalability testing tool. This has components which analyze the commute conditions in the application, a test engine which evaluates the scalability and a trace which identifies conflicting cache lines to improve scalability.
3. Implementing a Linux like POSIX based kernel, sv6 which caters to the bottlenecks detected by commuter and improves scalability.

Evaluations :
The authors used commuter to test the scalability for Linux OS. According to their tool, all the operations in Linux are supposed to commute and hence 100% scalable. But, in evaluation, they found that only 68% of Linux was scalable. This was because of the interface design aspects of the system calls and such. The OS sv6 has an improved version of these system calls, making use of the commute properties, making it 99% scalable.

What I found confusing :
If the commute rule helps improve scalability in multi core machines, how does this relate to distributed system settings? Has commuter been used there too?

Summary

Operating systems designers typically look for scalability opportunities at the implementation level. However, there is also potential to exploit scalability within system interfaces. This paper revisits the concept of commutability ( and suggests that whenever system interfaces are commutable they can also be implemented for scalability. The authors develop a model which determines commutability of an execution sequence, and uses this to demonstrate scalability in a generic Linux kernel compared to a variant which has been optimized for commutability.

Problem

Scalability has typically been thought of as an implementation problem, and any evaluations usually happen once a system is almost complete. This raises several problems: different architectures can present new bottlenecks, bottlenecks are harder to identify, and since the product is almost finished when testing begins, problems are impractical to fix.

Contributions

The main novel contribution in this paper is SIM (State-dependent, Interface-based, Monotonic) commutativity. This constrains commutativity to a certain system context, operation arguments and specific operations, however it also guarantees that when an operation commutes (execution operations can happen in any order) then an implementation exists that is conflict-free under these constraints. Additionally the authors develop a COMMUTER utility which generates tests of operations that may be able to commute and provide scalability improvements.

Evaluation

The authors use their COMMUTER tool to generate tests of operations that could be scalable; they use this tool to compare a standard Linux kernel to a variant called sv6 which has an optimized implementations of most POSIX calls. They demonstrate that standard Linux scales for 68% of the 13664 tests generated, and sv6 scales for 99% of all tests. next, they use two micro benchmarks along with a mail server to demonstrate performance benefits. This shows that a system using commutative APIs scales at almost an order of magnitude compared to a system that does not.

Confusions

What is the actual difference between scalability and performance? This paper seems to use these terms interchangeably but I thought there was some difference?

1. Summary
This paper introduces a new approach to designing scalable software - analyzing the scalability of interfaces. By designing an interface around commutable operations, it posits that the interface can be implemented in a manner that avoids memory-access conflicts. A relaxed, state-dependent version of commutativity is defined that exposes more latent commutativity in complex interfaces. A tool is designed that takes in a symbolic representation of an interface and generates its commutative operations and test cases to test those operations on any implementation of the interface.

2. Problem
Software scalability is usually assumed to be implementation-dependent. Software is checked for scalability at a late stage in the design pipeline. Scalability bottlenecks uncovered at that stage could be due to implementation-deficiencies or fundamental behaviors of the interface. The latter category cannot be fixed at such an advanced stage in the design. Instead, this paper proposes a method to build and prove an interface’s scalability even before it is implemented. Any future implementation of this interface can aspire to this model’s scalability as its goal.

3. Contributions
A common proxy to identify scalable code is the lack of cache-line access conflicts. If there is a conflict, the coherence protocol enforces serialization which hurts scalability. Commutative operations are usually scalable as the order of execution does not matter, hence avoiding communication/synchronization. This paper posits a rule that a communicable interface can be implemented in a conflict-free manner. Default commutativity can be too stringent for realistic interfaces. Instead, the paper introduces a state- & interface- dependent idea called SIM-communicativity. This conditions the commutativity of the interface to certain states and certain interface arguments/operations, opening up newer opportunities. Using the rule, the paper reasons about common POSIX interfaces and identifies common flaws that hurt scalability - for eg. compound operations within a single API, unnecessary ordering of operations etc. To automate the reasoning, they build a tool called COMMUTER. It contains 3 parts - the ANALYZER takes in a symbolic representation of the interface and generates commutative operations; the TESTGEN takes these commutative operations and generates test cases that can recreate those operations; the MTRACE will run the test cases on an implementation of the interface and identify memory access conflicts.

4. Evaluation
More scalable implementations of FS and Virtual memory interfaces are implemented on top of xv6 OS, called sv6. sv6 and Linux kernel v3.8 are tested using tests generated by COMMUTER. sv6 scales for ~99% of cases. sv6’s scalable interfaces also provide linear speedups with increasing core-counts.

5. Confusions
How is a specification of an interface constructed if all possible histories cannot be enumerated?

1. Summary

This paper introduces the scalable commutativity rule which helps to identify scalability opportunities in software interfaces and demonstrates the use of the rule in building scalable systems.

2. Problem

Developers typically improve the scalability of a system by first choosing a workload, evalutating the performance of that workload across a number of cores and identifying bottlenecks which are then worked on to rectify. However, this approach is limited as different workloads/cores exhibit new bottlenecks. Also, scalability is thought of as an implementation property. Thus, determining if certain operations can scale typically requires the developer to consider all possible implementations. This process is "difficult, unguided and does not scale itself".

3. Contributions

The Scalable Commutativity Rule
--------------------------------
Whenever operations commutes (no way to distinguish execution order through the interface), it can be implemented in a way that scales.
- A memory access is conflict-free if a core does not write to a cache line that is being read/written by another core.
- An operation scales if its implementation uses only conflict free memory access.
- Operations that commute are independent of each other. There is no communication between the operations. Thus, the memory access are conflict
free and the operations are scalable.
- Operations need to be SIM-commutative (state, interface, monotonic).
+ Makes reasoning about scalability possible w/o considering specific implementations, gives developers a clear scaling target

Examples of how existing POSIX interfaces can be changed to ensure operations commute are given. For example, compound operations returning multiple values should be broken down to return just a specific value so that there is less chance of operation memory conflicts. Also interfaces should not enforce unnecessary deterministic constraints (e.g creat - lowest available fd) and seek to not use strict ordering (e.g - messages sent via a local domain socket).

COMMUTER
--------------
Automates the process of spotting commutative interfaces and tests implementations for conflict freedom.
Analyzer
- Takes in a symbolic model of the behaviour of an interface and outputs the state, argument conditions under which the operations commute.
TestGen
- Produces test cases (in actual C code) of operations that should commute. Implementations can then be verified for scalability by testing against these test cases.The test cases consider both path and conflict coverage.
MTrace
- Runs the test cases on actual implementations. If a test fails, reports which variables are shared and what code accessed them.

4. Evaluation

COMMUTER was applied to a simplified model of POSIX covering 18 system calls. It generated 13664 test cases of operations that are commutative. In Linux however, only 9389 of these cases satisfied showing the lost opportunities for scaling.

However, it is not clear if optimizing for scalability will lead to a loss of performance in the sequential case. Hence, a new OS kernel (sv6) is implemented, guided by COMMUTER. In both the statbench and openbench microbenchmarks, the commutative versions of fstat and open scale linearly with the number of cores (upto 80) while the non-commutative interface is noted to limit scalability at even around 20 cores. In the application benchmark where a mail server is run, there is a collapse in performance in the non-commutative case whereas in the commutative case, there is a 7.5x scalability from 10 to 80 cores.

5. Confusions

I do not understand the monotonic requirement of SIM commutativty.

Summary
The problem of designing interfaces to scale with multicore processors has not been studied in incredible detailed because typically scalability is thought of as an implementation problem. In this paper, the authors present a method of identifying interfaces that can scale by identifying operations that can commute, or be executed in any order and not impact the outcome. They assert that when interface operations commute, they can be implemented in a way that scales. This is because their operations with be conflict-free, which means their memory accesses will not access the same data and limit scalability.
Problem
As the paper mentions, scalability is typically approached as an implementation problem. However, it is hard to design a system to be scalable when the only way of evaluating whether or not the system scales is to test it on benchmarks after the system has been built. To address this issue, the authors have implemented COMMUTER which automatically analyzes the interface and is able to identify the constraints on the implementation to make the operations conflict free. This assists the scalability in the creation of the interface and the implementation.
Contributions
One main contribution of the paper was their definition of SIM commutativity. They define SIM commutativity to be when actions may be executed in any order, but must maintain the order of the actions in a single thread. However, because the reorder of the actions among threads is allowed, commutativity is preserved. Additionally, the actions must be monotonic. This means any prefix of actions must not depend on future operations because depending on the future, the set of actions may not be commutative. Using this definition of SIM commutativity, the scalable commutativity rule can be generated. This states that if an implementation can generate an operation history where the history can commute, then the operations in that history are conflict-free. Because these operations are conflict free, the implementation will be able to scale well on a multicore processor. The paper also presents a number of suggestions for designing scalable interfaces, such as embracing non-determinism and lazily freeing resources. Both of these suggestions help scalability by limiting the number of shared data accesses the operations take. Additionally, the authors built a tool call COMMUTER that automatically analyzes an interfaces, generates test cases for that interface, and runs the test cases to confirm that the implementation is conflict-free.
Evaluation
To evaluate the effectiveness of new scalable implementations, the authors implemented an operating system called sv6 which has scalable implementations of many operations. Specifically, the authors tested their implementations against two microbenchmarks and one application benchmark. The two microbenchmarks tested were a scalable implementation of fstat and a scalable implementation of open. The stat benchmark scaled very well when all operations in the implementation commuted. Similarly, when the open implementation used the non-deterministic optimization, the implementation also scaled linearly.
Confusions
I was confused about the purpose of the CONTINUE action in the implementations. Also, what is the practicality of using the COMMUTER system? Aren’t there a lot of other considerations to take into account when designing interfaces? (usability, readability, etc).

Summary
This work is built around the proposed rule that the commutativity of interface operations guarantees the existence of a scalable implementation. The authors prove this scalable commutativity rule using formal methods, and present an automated tool called Commuter to analyze the scalability of an implementation.

Problem
The authors claim that reasoning about the scalability of multicore software is a tough problem. Identifying bottlenecks through various profiling techniques doesn’t help understand if the limitation is implementation-dependent or inherent in the interface/task/hardware environment itself. The authors also aim to associate scalability with the interface design.

Contributions
The main contribution of this work is the application of the association between commutativity and concurrency to the formulation of the scalable commutativity rule. SIM (state-independent, interface-based, monotonic) commutativity is introduced, which adds constraints to commutativity in order to better correlate it with scalability, or rather conflict freedom. These arguments are formally developed and proved in the paper. The authors apply these reasonings to discuss the scalability of some POSIX APIs, and share high-level rules to enhance the scalability of the POSIX interfaces. Finally, a high-level tool is described, to automate the application of this process to real-world APIs.

Evaluation
The authors evaluate the effectiveness of the Commuter tool by comparing the reasonably scalable implementations of standard POSIX APIs as found in Linux with similar APIs implemented in their own OS kernel called sv6. Test cases are generated by the Commuter tool to evaluate the scalability of various system calls in both Linux and sv6. The sv6 implementations were built guided by the observations offered by the Analyzer sub-module of Commuter. This results in the improvement that system call interfaces in sv6 are scalable in 99% of cases tested up from the 70% in unmodified Linux.

Confusions
Could you explain how the “read” operation of a state component is determined and what the component replacement means?