:authors: Jason Lowe-Power

Part II: Modifying and extending gem5
=====================================

Setting up development environment
----------------------------------

* Use the style guide
* Install the style guide
* Use mercurial queues (or git or whatever)

--------------------------------------

Simple SimObject
----------------
To come

--------------------------------------

Debugging gem5
--------------
To come

--------------------------------------

Event-driven programming
------------------------

* Add an event wrapper to the HelloObject from last chapter.
* Add a processEvent function

hello_object.hh
~~~~~~~~~~~~~~~
.. code-block:: c++

      private:
        void processEvent();

        EventWrapper<HelloObject, &HelloObject::processEvent> event;

* Initialize the event
* Implement the processEvent function

hello_object.cc
~~~~~~~~~~~~~~~
.. code-block:: c++

    HelloObject::HelloObject(HelloObjectParams *params) :
        SimObject(params), event(*this)

    void
    HelloObject::processEvent()
    {
        DPRINTF(Hello, "Hello world! Processing the event!\n");
    }

* Add a startup function to the header
* Schedule an event

hello_object.hh
~~~~~~~~~~~~~~~
.. code-block:: c++

    void startup();

hello_object.cc
~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    HelloObject::startup()
    {
        schedule(event, 100);
    }

* Recompile and run gem5

--------------------------------------

* Add two parameters to class: latency, timesLeft

hello_object.hh
~~~~~~~~~~~~~~~
.. code-block:: c++

        Tick latency;

        int timesLeft;

* Initialize these parameters

hello_object.cc
~~~~~~~~~~~~~~~
.. code-block:: c++

    HelloObject::HelloObject(HelloObjectParams *params) :
        SimObject(params), event(*this), latency(100), timesLeft(10)

* update startup and process event

hello_object.cc
~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    HelloObject::startup()
    {
        schedule(event, latency);
    }

    void
    HelloObject::processEvent()
    {
        timesLeft--;
        DPRINTF(Hello, "Hello world! Processing the event! %d left\n", timesLeft);

        if (timesLeft <= 0) {
            DPRINTF(Hello, "Done firing!\n");
        } else {
            schedule(event, curTick() + latency);
        }
    }


--------------------------------------

Adding parameters
-----------------

* Talk about simple parameters

HelloObject.py
~~~~~~~~~~~~~~
.. code-block:: python

    class HelloObject(SimObject):
        type = 'HelloObject'
        cxx_header = "learning_gem5/hello_object.hh"

        time_to_wait = Param.Latency("Time before firing the event")
        number_of_fires = Param.Int(1, "Number of times to fire the event before "
                                       "goodbye")

* Update the constructor

hello_object.cc
~~~~~~~~~~~~~~~
.. code-block:: c++

    HelloObject::HelloObject(HelloObjectParams *params) :
        SimObject(params),
        event(*this),
        myName(params->name),
        latency(params->time_to_wait),
        timesLeft(params->number_of_fires)
    {
        DPRINTF(Hello, "Created the hello object with the name %s\n", myName);
    }

* Run gem5 without updating the config file and get an error
* Fix the above error

run_hello.py
~~~~~~~~~~~~
.. code-block:: python

    root.hello = HelloObject(time_to_wait = '2us')

* or

.. code-block:: python

    root.hello = HelloObject()
    root.hello.time_to_wait = '2us'

* Run again
* Modify config to fire more than once

* Run again

.. code-block:: python

    root.hello.number_of_fires = 10

----------------------------------------------

* Clear changes, add 2 patches to hello-4-goodbye

* Add a goodbye object to the python file
* Talk about what this object is going to do. It has a buffer that it fills at some maximum rate.

HelloObject.py
~~~~~~~~~~~~~~
.. code-block:: python

    class GoodbyeObject(SimObject):
        type = 'GoodbyeObject'
        cxx_header = "learning_gem5/goodbye_object.hh"

        buffer_size = Param.MemorySize('1kB',
                                       "Size of buffer to fill with goodbye")
        write_bandwidth = Param.MemoryBandwidth('100MB/s', "Bandwidth to fill "
                                                "the buffer")

* Show "sayGoodbye" interface

goodbye_object.hh
~~~~~~~~~~~~~~~~~
.. code-block:: c++

    void sayGoodbye(std::string name);

* Implement sayGoodbye

goodbye_object.cc
~~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    GoodbyeObject::sayGoodbye(std::string other_name)
    {
        DPRINTF(Hello, "Saying goodbye to %s\n", other_name);

        message = "Goodbye " + other_name + "!! ";

        fillBuffer();
    }

* Implement missing fillBuffer

goodbye_object.cc
~~~~~~~~~~~~~~~~~
.. code-block:: c++

    if (bufferUsed < bufferSize - 1) {
        // Wait for the next copy for as long as it would have taken
        DPRINTF(Hello, "Scheduling another fillBuffer in %d ticks\n",
                bandwidth * bytes_copied);
        schedule(event, curTick() + bandwidth * bytes_copied);
    } else {
        DPRINTF(Hello, "Goodbye done copying!\n");
        // Be sure to take into account the time for the last bytes
        exitSimLoop(buffer, 0, curTick() + bandwidth * bytes_copied);
    }

* Make changes to hello object

HelloObject.py
~~~~~~~~~~~~~~
.. code-block:: python

    goodbye_object = Param.GoodbyeObject("A goodbye object")

hello_object.hh
~~~~~~~~~~~~~~~
.. code-block:: c++

        GoodbyeObject& goodbye;

hello_object.cc
~~~~~~~~~~~~~~~
.. code-block:: c++

    if (timesLeft <= 0) {
        DPRINTF(Hello, "Done firing!\n");
        goodbye.sayGoodbye(myName);

* Update the config script

run_hello.py
~~~~~~~~~~~~
.. code-block:: python

    root.hello.goodbye_object = GoodbyeObject(buffer_size='100B')

* Run gem5
* Make other changes for examples

run_hello.py
~~~~~~~~~~~~
.. code-block:: python

    root.hello = HelloObject(time_to_wait = '2us', number_of_fires = 5)
    root.hello2 = HelloObject(time_to_wait = '2.5us', number_of_fires = 3)

    root.goodbye = GoodbyeObject(buffer_size = '30B')
    root.goodbye2 = GoodbyeObject(buffer_size = '30B')

    root.hello.goodbye_object = root.goodbye2
    root.hello2.goodbye_object = root.goodbye


* Run gem5

----------------------

MemObjects
----------

* Show slides about master/slave and packets
* packets
  * Request (addr, requestor)
  * command (can change)
  * size
  * data (pointer)
* port interface

.. figure:: ../_static/figures/master_slave_1.png
    :width: 40 %

    Simple master-slave interaction when both can accept the request and the response.

.. figure:: ../_static/figures/master_slave_2.png
    :width: 40 %
    :alt: Slave busy interaction

    Simple master-slave interaction when the slave is busy

.. figure:: ../_static/figures/master_slave_3.png
   :width: 40 %
   :alt: Master busy interaction

   Simple master-slave interaction when the master is busy

--------------------------

* Make the SConscript file

Sconscript
~~~~~~~~~~
.. code-block:: python

    Import('*')

    SimObject('SimpleMemobj.py')
    Source('simple_memobj.cc')

    DebugFlag('SimpleMemobj')

* Create a simple memobj description

SimpleMemobj.py
~~~~~~~~~~~~~~~
.. code-block:: python

    from m5.params import *
    from m5.proxy import *
    from MemObject import MemObject

    class SimpleMemobj(MemObject):
        type = 'SimpleMemobj'
        cxx_header = "learning_gem5/simple_memobj/simple_memobj.hh"

        inst_port = SlavePort("CPU side port, receives requests")
        data_port = SlavePort("CPU side port, receives requests")
        mem_side = MasterPort("Memory side port, sends requests")

* Define the header file
* Point out "public MemObject"

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    #ifndef __LEARNING_GEM5_SIMPLE_MEMOBJ_SIMPLE_MEMOBJ_HH__
    #define __LEARNING_GEM5_SIMPLE_MEMOBJ_SIMPLE_MEMOBJ_HH__

    #include "mem/mem_object.hh"
    #include "params/SimpleMemobj.hh"

    class SimpleMemobj : public MemObject
    {
      private:

      public:

        /** constructor
         */
        SimpleMemobj(SimpleMemobjParams *params);
    };
    #endif

* Define the CPU-side slave port
* Talk about each of the functions below

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class CPUSidePort : public SlavePort
    {
      private:
        SimpleMemobj *owner;

      public:
        CPUSidePort(const std::string& name, SimpleMemobj *owner) :
            SlavePort(name, owner), owner(owner)
        { }

        AddrRangeList getAddrRanges() const override;

      protected:
        Tick recvAtomic(PacketPtr pkt) override { panic("recvAtomic unimpl."); }
        void recvFunctional(PacketPtr pkt) override;
        bool recvTimingReq(PacketPtr pkt) override;
        void recvRespRetry() override;
    };

* define the memory side master port
* Talk about each of the functions below

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class MemSidePort : public MasterPort
    {
      private:
        SimpleMemobj *owner;

      public:
        MemSidePort(const std::string& name, SimpleMemobj *owner) :
            MasterPort(name, owner), owner(owner)
        { }

      protected:
        bool recvTimingResp(PacketPtr pkt) override;
        void recvReqRetry() override;
        void recvRangeChange() override;
    };

* Define the MemObject interface

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class SimpleMemobj : public MemObject
    {
      private:

        <CPUSidePort declaration>
        <MemSidePort declaration>

        CPUSidePort instPort;
        CPUSidePort dataPort;

        MemSidePort memPort;

      public:
        SimpleMemobj(SimpleMemobjParams *params);

        BaseMasterPort& getMasterPort(const std::string& if_name,
                                      PortID idx = InvalidPortID) override;

        BaseSlavePort& getSlavePort(const std::string& if_name,
                                    PortID idx = InvalidPortID) override;

    };

* Initialize things in construcutor

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    SimpleMemobj::SimpleMemobj(SimpleMemobjParams *params) :
        MemObject(params),
        instPort(params->name + ".inst_port", this),
        dataPort(params->name + ".data_port", this),
        memPort(params->name + ".mem_side", this),
    {
    }

* Implement getMasterPort

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    BaseMasterPort&
    SimpleMemobj::getMasterPort(const std::string& if_name, PortID idx)
    {
        if (if_name == "mem_side") {
            return memPort;
        } else {
            return MemObject::getMasterPort(if_name, idx);
        }
    }

* Implement getSlavePort

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    BaseSlavePort&
    SimpleMemobj::getSlavePort(const std::string& if_name, PortID idx)
    {
        if (if_name == "inst_port") {
            return instPort;
        } else if (if_name == "data_port") {
            return dataPort;
        } else {
            return MemObject::getSlavePort(if_name, idx);
        }
    }

* Pass through some of the functions for CPU side port

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    AddrRangeList
    SimpleMemobj::CPUSidePort::getAddrRanges() const
    {
        return owner->getAddrRanges();
    }

    void
    SimpleMemobj::CPUSidePort::recvFunctional(PacketPtr pkt)
    {
        return owner->handleFunctional(pkt);
    }

    void
    SimpleMemobj::handleFunctional(PacketPtr pkt)
    {
        memPort.sendFunctional(pkt);
    }

    AddrRangeList
    SimpleMemobj::getAddrRanges() const
    {
        DPRINTF(SimpleMemobj, "Sending new ranges\n");
        return memPort.getAddrRanges();
    }

* Pass through some of the functions for Mem side port

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::MemSidePort::recvRangeChange()
    {
        owner->sendRangeChange();
    }

    void
    SimpleMemobj::sendRangeChange()
    {
        instPort.sendRangeChange();
        dataPort.sendRangeChange();
    }

---------------------------------------------

* NOw the fun part. Implementing the send/receives
* Let's start with receive

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::CPUSidePort::recvTimingReq(PacketPtr pkt)
    {
        if (!owner->handleRequest(pkt)) {
            needRetry = true;
            return false;
        } else {
            return true;
        }
    }

* Now, we need to do handle request

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::handleRequest(PacketPtr pkt)
    {
        if (blocked) {
            return false;
        }
        DPRINTF(SimpleMemobj, "Got request for addr %#x\n", pkt->getAddr());
        blocked = true;
        memPort.sendPacket(pkt);
        return true;
    }

* Let's add a convienency function in the memside port

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::MemSidePort::sendPacket(PacketPtr pkt)
    {
        panic_if(blockedPacket != nullptr, "Should never try to send if blocked!");
        if (!sendTimingReq(pkt)) {
            blockedPacket = pkt;
        }
    }

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class MemSidePort : public MasterPort {
        PacketPtr blockedPacket;
      public:
        void sendPacket(PacketPtr pkt);

* Implement code to handle retries

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::MemSidePort::recvReqRetry()
    {
        assert(blockedPacket != nullptr);

        PacketPtr pkt = blockedPacket;
        blockedPacket = nullptr;

        sendPacket(pkt);
    }

---------------------------------------------------------------

* Implement the code for receiving requests

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::CPUSidePort::recvTimingReq(PacketPtr pkt)
    {
        if (!owner->handleRequest(pkt)) {
            needRetry = true;
            return false;
        } else {
            return true;
        }
    }

* Add variable to remember when we need to send the CPU a retry

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class CPUSidePort : public SlavePort
    {
        bool needRetry;

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::handleResponse(PacketPtr pkt)
    {
        assert(blocked);
        DPRINTF(SimpleMemobj, "Got response for addr %#x\n", pkt->getAddr());

        blocked = false;

        // Simply forward to the memory port
        if (pkt->req->isInstFetch()) {
            instPort.sendPacket(pkt);
        } else {
            dataPort.sendPacket(pkt);
        }

        return true;
    }

* Now, we need the convience function to send packets

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class CPUSidePort : public SlavePort
    {
        PacketPtr blockedPacket;
      public:
        void sendPacket(PacketPtr pkt);

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::CPUSidePort::sendPacket(PacketPtr pkt)
    {
        panic_if(blockedPacket != nullptr, "Should never try to send if blocked!");

        if (!sendTimingResp(pkt)) {
            blockedPacket = pkt;
        }
    }

* Implement recvRespRetry

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::CPUSidePort::recvRespRetry()
    {
        assert(blockedPacket != nullptr);

        PacketPtr pkt = blockedPacket;
        blockedPacket = nullptr;

        sendPacket(pkt);
    }

* Implement trySendRetry

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class CPUSidePort : public SlavePort {
        void trySendRetry();

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::CPUSidePort::trySendRetry()
    {
        if (needRetry && blockedPacket == nullptr) {
            needRetry = false;
            DPRINTF(SimpleMemobj, "Sending retry req for %d\n", id);
            sendRetryReq();
        }
    }


simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    SimpleMemobj::handleResponse(PacketPtr pkt)
    {
        instPort.trySendRetry();
        dataPort.trySendRetry();

-----------------------------------

* Update simple config file

simple.py
~~~~~~~~~
.. code-block:: python

    system.cpu = TimingSimpleCPU()

    system.memobj = SimpleMemobj()

    system.cpu.icache_port = system.memobj.inst_port
    system.cpu.dcache_port = system.memobj.data_port

    system.membus = SystemXBar()

    system.memobj.mem_side = system.membus.slave

* Run simple.py

---------------------------------------------

Making a cache
--------------

* Add parameters to memobj

SimpleMemobj.py
~~~~~~~~~~~~~~~
.. code-block:: python

    latency = Param.Cycles(1, "Cycles taken on a hit or to resolve a miss")

    size = Param.MemorySize('16kB', "The size of the cache")

    system = Param.System(Parent.any, "The system this cache is part of")

* Talk about the parent.any proxy parameter

* Add latency/size/system to constructor

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    latency(params->latency),
    blockSize(params->system->cacheLineSize()),
    capacity(params->size / blockSize),

* Implement new "handleRequest"

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::handleRequest(PacketPtr pkt, int port_id)
    {
        if (blocked) {
            return false;
        }
        DPRINTF(SimpleMemobj, "Got request for addr %#x\n", pkt->getAddr());

        blocked = true;
        waitingPortId = port_id;

        schedule(new AccessEvent(this, pkt), clockEdge(latency));

        return true;
    }

* Talk about the clockEdge function and clocked-objects

* Implement the access event

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    class AccessEvent : public Event
    {
      private:
        SimpleMemobj *cache;
        PacketPtr pkt;
      public:
        AccessEvent(SimpleMemobj *cache, PacketPtr pkt) :
            Event(Default_Pri, AutoDelete), cache(cache), pkt(pkt)
        { }
        void process() override {
            cache->accessTiming(pkt);
        }
    };

* Implement the accessTiming function

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void accessTiming(PacketPtr pkt);

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::accessTiming(PacketPtr pkt)
    {
        bool hit = accessFunctional(pkt);
        if (hit) {
            pkt->makeResponse();
            sendResponse(pkt);
        } else {
            <miss handling>
        }
    }

* Note; It's a good idea to separate out functional from timing functions
* Miss handling is complicated by the block size

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::accessTiming(PacketPtr pkt)
    {
        bool hit = accessFunctional(pkt);
        if (hit) {
            pkt->makeResponse();
            sendResponse(pkt);
        } else {
            Addr addr = pkt->getAddr();
            Addr block_addr = pkt->getBlockAddr(blockSize);
            unsigned size = pkt->getSize();
            if (addr == block_addr && size == blockSize) {
                DPRINTF(SimpleMemobj, "forwarding packet\n");
                memPort.sendPacket(pkt);
            } else {
                DPRINTF(SimpleMemobj, "Upgrading packet to block size\n");
                panic_if(addr - block_addr + size > blockSize,
                         "Cannot handle accesses that span multiple cache lines");

                assert(pkt->needsResponse());
                MemCmd cmd;
                if (pkt->isWrite() || pkt->isRead()) {
                    cmd = MemCmd::ReadReq;
                } else {
                    panic("Unknown packet type in upgrade size");
                }

                PacketPtr new_pkt = new Packet(pkt->req, cmd, blockSize);
                new_pkt->allocate();

                outstandingPacket = pkt;

                memPort.sendPacket(new_pkt);
            }
        }
    }

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    PacketPtr outstandingPacket;

* Update handle response to be able to accept responses from the upgraded packets

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::handleResponse(PacketPtr pkt)
    {
        assert(blocked);
        DPRINTF(SimpleMemobj, "Got response for addr %#x\n", pkt->getAddr());
        insert(pkt);

        if (outstandingPacket != nullptr) {
            accessFunctional(outstandingPacket);
            outstandingPacket->makeResponse();
            delete pkt;
            pkt = outstandingPacket;
            outstandingPacket = nullptr;
        } // else, pkt contains the data it needs

        sendResponse(pkt);

        return true;
    }

-------------------------------------------------------

* Implementing the functional cache logic, now.

simple_memobj.hh
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void insert(PacketPtr pkt);
    bool accessFunctional(PacketPtr pkt);
    std::unordered_map<Addr, uint8_t*> cacheStore;

* Implement the access logic

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    bool
    SimpleMemobj::accessFunctional(PacketPtr pkt)
    {
        Addr block_addr = pkt->getBlockAddr(blockSize);
        auto it = cacheStore.find(block_addr);
        if (it != cacheStore.end()) {
            if (pkt->isWrite()) {
                pkt->writeDataToBlock(it->second, blockSize);
            } else if (pkt->isRead()) {
                pkt->setDataFromBlock(it->second, blockSize);
            } else {
                panic("Unknown packet type!");
            }
            return true;
        }
        return false;
    }

* Implement the insert logic

simple_memobj.cc
~~~~~~~~~~~~~~~~
.. code-block:: c++

    void
    SimpleMemobj::insert(PacketPtr pkt)
    {
        if (cacheStore.size() >= capacity) {
            // Select random thing to evict. This is a little convoluted since we
            // are using a std::unordered_map. See http://bit.ly/2hrnLP2
            int bucket, bucket_size;
            do {
                bucket = random_mt.random(0, (int)cacheStore.bucket_count() - 1);
            } while ( (bucket_size = cacheStore.bucket_size(bucket)) == 0 );
            auto block = std::next(cacheStore.begin(bucket),
                                   random_mt.random(0, bucket_size - 1));

            RequestPtr req = new Request(block->first, blockSize, 0, 0);
            PacketPtr new_pkt = new Packet(req, MemCmd::WritebackDirty, blockSize);
            new_pkt->dataDynamic(block->second); // This will be deleted later

            DPRINTF(SimpleMemobj, "Writing packet back %s\n", pkt->print());
            memPort.sendTimingReq(new_pkt);

            cacheStore.erase(block->first);
        }
        uint8_t *data = new uint8_t[blockSize];
        cacheStore[pkt->getAddr()] = data;

        pkt->writeDataToBlock(data, blockSize);
    }

---------------------------------------------

* update the config file

simple.py
~~~~~~~~~
.. code-block:: python

    system.memobj = SimpleMemobj(size='1kB')

* Run it!