adding ai_economist for modding

2023-01-12 16:41:38 +01:00
parent 0479a4f6a4
commit f177f8f0ba
85 changed files with 19373 additions and 2 deletions
@@ -0,0 +1,120 @@
+# Real Business Cycle (RBC)
+This directory implements a **Real-Business-Cycle** (RBC) simulation with many heterogeneous, interacting strategic agents of various types, such as **consumers, firms, and the government**. For details, please refer to this paper "Finding General Equilibria in Many-Agent Economic Simulations using Deep Reinforcement Learning (ArXiv link forthcoming)". We also provide training code that uses deep multi-agent reinforcement learning to determine optimal economic policies and dynamics in these many agent environments. Below are instructions required to launch the training runs.
+
+**Note: The experiments require a GPU to run!**
+
+## Dependencies
+
+- torch>=1.9.0
+- pycuda==2021.1
+- matplotlib==3.2.1
+
+## Running Local Jobs
+To run a hyperparameter sweep of jobs on a local machine, use (see file for command line arguments and hyperparameter sweep dictionaries)
+
+```
+python train_multi_exps.py
+```
+
+## Configuration Dictionaries
+
+Configuration dictionaries are currently specified in Python code, and then written as `hparams.yaml` in the job directory. For examples, see the file `constants.py`. The dictionaries contain "agents", "world", and "train" dictionaries which contain various hyperparameters.
+
+## Hyperparameter Sweeps
+
+The files `train_multi_exps.py` allow hyperparameter sweeps. These are specified in `*_param_sweeps` dictionaries in the file. For each hyperparameter, specify a list of one or more choices. The Cartesian product of all choices will be used.
+
+## Approximate Best Response Training
+
+To run a single approximate best-response (BR) training job on checkpoint policies, run `python train_bestresponse.py ROLLOUT_DIR NUM_EPISODES_TO_TRAIN --ep-strs ep1 ep2 --agent-type all`. The `--ep-strs` argument specifies which episodes to run on (for example, policies from episode 0, 10000, and 200000). These must be episodes for which policies were saved. It is possible to specify a single agent type.
+
+
+## What Will Be Saved?
+
+A large amount of data will be saved -- one can set hyperparamter `train.save_dense_every` in the configuration dictionary (`hparams.yaml`/`constants.py`) to reduce this.
+
+At the top level, an experiment directory stores the results of many runs in a hyperparameter sweep. Example structure:
+
+```
+experiment/experimentname/
+    rollout-999999-99999/
+        brconsumer/
+            ...
+        brfirm/
+            episode_XXXX_consumer.npz
+            episode_XXXX_government.npz
+            episode_XXXX_firm.npz
+            saved_models/
+                consumer_policy_XXX.pt
+                firm_policy_XXX.pt
+                government_policy_XXX.pt.
+        brgovernment/
+            ...
+        hparams.yaml
+        action_arrays.pickle
+        episode_XXXX_consumer.npz
+        episode_XXXX_government.npz
+        episode_XXXX_firm.npz
+        saved_models/
+            consumer_policy_XXX.pt
+            firm_policy_XXX.pt
+            government_policy_XXX.pt.
+
+    rollout-777777-77777/
+        ...
+```
+
+Files:
+
+`rollout-XXXXXX-XXX`: subdirectory containing all output for a single run.
+
+`hparams.yaml`: configuration dictionary with hyperparameters
+
+`action_arrays.pickle`: contains saved action arrays (allowing mapping action indices to the actual action, e.g. index 1 is price 1000.0, etc.)
+
+`episode_XXXX_AGENTTYPE.npz`: Contains dense rollouts stored as the output of a numpy.savez call. When loaded, can be treated like a dictionary of numpy arrays.  Has keys: `['states', 'actions', 'rewards', 'action_array', 'aux_array']` (view keys by using `.files`). `states`, `actions`, `rewards`, and `aux_array` all refer to saved copies of CUDA arrays (described below). `action_array` is a small array mapping action indices to the actual action.
+
+`saved_models/AGENTTYPE_policy_XXX.pt`: a saved PyTorch state dict of the policy network, after episode XXX.
+
+## Structure Of Arrays
+
+`states` for any given agent type is an array storing observed states. It has shape `batch_size, ep_length, num_agents, agent_total_state_dim`.
+
+`actions` is an array consisting of the action _indices_ (integers). For firms and government, it is of shape `batch_size, ep_length, num_agents`. For consumers, it is of shape `batch_size, ep_length, num_agents, num_action_heads`.
+
+`rewards` stores total rewards, and is of shape `batch_size, ep_length, num_agents`.
+
+The `aux_array` stores additional information and may differ per agent type. The consumer `aux_array` stores _actual_ consumption of each firm's good (as opposed to attempted consumption). The firm `aux_array` stores the amount bought by the export market.
+
+## State Array Layout:
+
+States observed by each agent consist of a global state, plus additional state dimensions per agent.
+
+Global state: total dimension 4 * num_firms + 2 + 1
+- prices: 1 per firm
+- wages: 1 per firm
+- inventories: 1 per firm
+- overdemanded flag: 1 per firm
+- time
+
+Consumer additional state variables: total dimension global state + 2
+- budget
+- theta
+
+Firm additional state variables: total dimension global state + 3 + num_firms
+- budget
+- capital
+- production alpha
+- one-hot representation identifying which firm
+
+## What Gets Loaded And Written By BR Code?
+
+The best response code loads in the `hparams.yaml` file, and the policies at a given time step (i.e. `saved_models/...policy_XXX.pt`). It then trains one of the policies while keeping the others fixed. Results are written to directories `brfirm`, `brconsumer`, `brgovernment` and contain dense rollouts and saved policy checkpoints, but from the best response training.
+
+## Which Hyperparameters Are Managed And Where?
+
+Initial values of state variables (budgets, initial wages, levels of capital, and so on) are set by the code in the method `__init_cuda_data_structs`. Some of these can be controlled from the hyperparameter dict; others are currently hardcoded.
+
+Other hyperparameters are specified in the configuration dictionary.
+
+Finally, the technology parameter (A) of the production function is currently hardcoded in the function call in `rbc/cuda/firm_rbc.cu`.