Skip to main content

Describing Experimental/Simulation Setup


Sometimes the results of a performance analysis may depend on the computers used and/or specific features of software/libraries. In such cases it is extremely important to describe the experimental/simulation set up in details. It enables others to repeat those experiments as well as check whether the results are rigorous, statistically sound, and unbiased. Unfortunately, "Simulation Setup" is the shortest section in many research papers where authors try to save space by cutting down as much as details.
Here are some tips on what to include (in addition to describing the experimental/simulation setup) based on my experiences:
Type of Simulation
  • Are the results based on Experimentation, Emulation, or Simulation? If simulations also mention further details like whether it is Discrete Event, Montacarlo, Stochastic, or Deterministic simulation.
  •  Are the results for Steady, Dynamic, Starting/ramp-up, or Terminating state(s)?
Number of Experiments/Simulations
  • No of samples
  • Sampling methods
  • Confidence intervals, accuracy levels
  • Random number generation, whether a different seed is used for every run, what seeds were used (particularly if you use a widely available simulator/emulator)
Computer(s) Used
  • CPU - Clock speed, no of cores, cache sizes, CPU model (specific model no in addition to saying, e.g., Intel I7, Xeon, Itanium or AMD Opteron), and special instructions used (e.g., MMX, SSE, AVX)
  • Memory - capacity, speed
  • Accelerators used, e.g., GPUs or Intel Phi (only when they have an impact)
  • Sensors and actuators attached (only when they have an impact)
  • Operating system - type, version, special features, other services that were running, load on the node, performance enhancements/optimizations
  • Network connections - link speed, type of connection (e.g., Ethernet, WiFi, Bluetooth, ADSL, Dial-up, T1, 3G, HSPA+, 4G, etc.), latency, intermediate nodes (firewalls, proxies), etc. (only when they have an impact)
  • Cluster setup - interconnection network (e.g., ring, torus, mesh), type of connections (e.g., Ethernet or Infiniband) (only when they have an impact)
Applications/Simulators/Emulators/Libraries
  • A detailed list of tools used, e.g., math libraries, topology generators, workload generators, etc.
  • Their version numbers, what specific features/options were enabled/disabled
  • Specific parameters, e.g., packet/workload generation model, packet loss rate, packet size, inter-packet gap, job arrival time distribution, probability that one node connects to another, node churn rate, etc.
  • Compiler and compiler options, e.g., gcc, g++, Intel compilers, with/without optimizations, special flags
  • Cluster middleware (only when they have an impact)

Comments

Post a Comment

Popular posts from this blog

1. Building P2P Simulators with OverSim - Where to Begin

This could be a series of blog posts about extending or developing your own OverSim applications & overlay networks. OverSim has a minimal tutorial on writing your own application & overlay network; however, it doesn't show the big picture. So, I'm wasting lots of time playing with code & trying to understand the rest. Good thing is, I like it more & more as I understand. You need to change/develop only a few things, but finding out which ones is a hell of a task. I hope this will not only make my life easy but also will be useful to new comers. Here's what you need to do: You need some background on OMNeT++ OverSim extend OMNeT++. But sometime it has its own way of doing things (to make your task even simple) so understand the differences . Develop several OMNeT++ simulators. TicToc is a good one to start with. Extend it as you imagine. Read Towards a Common API for Structured Peer-to-Peer Overlays , which is the basis for OvseSim's AP
41 comments
Read more

Distributed Systems Mind Map

Though distributed systems are inherently distributed, autonomous, and parallel, all textbooks that I have read so far are boring & serial, going from one topic to another as they are independent topics. Wondering about how best to present all related topics and their relationships in my class, I came up with the following mind map. I was able to explain most of the key concepts using a single example of a web-based business that started selling flowers. It was a student who suggested that we consider selling flowers. It turned out to be a good example, as there had to be a connection to the physical world where the business also needed a geographical distributed delivery system. Taught of sharing the mind map. Though most of those branches can be broken down further this level of detailed was sufficient for my class (by the way it took 1 hour to finish the discussion on this slide). Distributed Systems Mind Map
1 comment
Read more