Team 5 shadowdash demo

[qianxichen233]

This PR is a demo of shadowdash from team5. A lot of things aren't cleaned up yet, so it might be a little messy here currently. The purpose of this PR is to provide a runnable shadowdash demo for people to play around and experiment with, and perhaps fix current issues in this shadowdash demo.

How to run the demo:

1. First, let's create a libmanifest.so:

• navigate to convertor folder: cd convertor
• compile manifest: cd manifest && ./compile.sh && cd ../
• convert a ninja file (e.g. zlib) to compile later:python3 convertor.py ./test_ninja_build/zlib_build.ninja
• compile the manifest to libmanifest.so: cd output && ./compile.sh

2. Then, build shadowdash:

• navigate to shadowdash root
• You probably need to replace the absolute library path in CMakeLists.txt at line 220 to point to the output folder. (This should be fixed later, but for the demo, let's just do this temporarily)
• run cmake -Bbuild-cmake && cmake --build build-cmake

3. run shadowdash:

• goto the root folder of the software you want to build (e.g. zlib)
• run env LD_LIBRARY_PATH="$SHADOWDASH_ROOT/convertor/output" $SHADOWDASH_ROOT/build-cmake/shadowdash all

NOTES:

1. default and pool syntax hasn't been added to this demo yet. We do have designed the default and pool in new language and it should be easy to add them to the demo. But again, since this is a demo and these two syntax is not very important to the building itself, so it hasn't been added yet. One consequence is that the build target must be manually specified when run the shadowdash (e.g. shadowdash all instead of just shadowdash).
2. This version of demo just experimented with pre-defining string variables, which significantly reduced the compilation time from ~3min to 40 seconds for LLVM.
3. The rule object is currently still defined within a single file, which causing it taking ~15 seconds to compile. This can be refactored and split the rule objects into multiple files just like build objects. But since 15 seconds is still not the biggest overhead of current manifest compilation, so this could be implemented later once the biggest overhead (sub-manifest compilation) is addressed.
4. metric endpoint is already added for shadowdash for benchmark purpose. by running ./shadowdash all -d stats -n (-n is optional but recommended), the time spent on running manifest can be shown. If you want to measure the time multiple times, a python script (bench.py under convertor folder) is already provided to do this. The script currently does not take any argument, so you probably need to adjust the parameter at the beginning of the script to make it run sccessfully.
5. Under convertor/convertors folder, you may see there are a lot of different versions of convertors. The most up to date version is convertor_split_func_new.py, which is already soft-linked by convertor/convertor.py.
6. There are also 3 optional parameters that can be adjust at the beginning of the convertor script.

• blocks means splitting the manifest into how many blocks
• optimizer specifies the c++ optimizer (O0, O1, O2, etc)
• job_threads specifies using how many threads to build sub-manifests.

[qianxichen233]

@kyotov Hey, professor, here is an shadowdash demo that you can play around. Currently I have already implemented part of the suggestions you gave last time: 1. concatenating literal strings 2. pre-define all the strings 3. refactor manifest.h and put the function definition into another file
The compilation speed does improve a lot by these suggestions (primarily from 2nd suggestion), and the manifest of LLVM can be compiled within 40 seconds with O0. (or 2 min with O1, 5min with O3). However, the runtime speed does not improve a lot.
I am currently thinking about using just references for all the strings and potentially other objects, which should reduce a lot of object copying overhead. Since now all the strings are pre-defined and live in the global scope, so they should not be destructed anymore and we can just reference them. I havn't implemented this yet, so this might be my next goal.