121 lines
5.9 KiB
Markdown
121 lines
5.9 KiB
Markdown
# Real Business Cycle (RBC)
|
|
This directory implements a **Real-Business-Cycle** (RBC) simulation with many heterogeneous, interacting strategic agents of various types, such as **consumers, firms, and the government**. For details, please refer to this paper "Finding General Equilibria in Many-Agent Economic Simulations using Deep Reinforcement Learning (ArXiv link forthcoming)". We also provide training code that uses deep multi-agent reinforcement learning to determine optimal economic policies and dynamics in these many agent environments. Below are instructions required to launch the training runs.
|
|
|
|
**Note: The experiments require a GPU to run!**
|
|
|
|
## Dependencies
|
|
|
|
- torch>=1.9.0
|
|
- pycuda==2021.1
|
|
- matplotlib==3.2.1
|
|
|
|
## Running Local Jobs
|
|
To run a hyperparameter sweep of jobs on a local machine, use (see file for command line arguments and hyperparameter sweep dictionaries)
|
|
|
|
```
|
|
python train_multi_exps.py
|
|
```
|
|
|
|
## Configuration Dictionaries
|
|
|
|
Configuration dictionaries are currently specified in Python code, and then written as `hparams.yaml` in the job directory. For examples, see the file `constants.py`. The dictionaries contain "agents", "world", and "train" dictionaries which contain various hyperparameters.
|
|
|
|
## Hyperparameter Sweeps
|
|
|
|
The files `train_multi_exps.py` allow hyperparameter sweeps. These are specified in `*_param_sweeps` dictionaries in the file. For each hyperparameter, specify a list of one or more choices. The Cartesian product of all choices will be used.
|
|
|
|
## Approximate Best Response Training
|
|
|
|
To run a single approximate best-response (BR) training job on checkpoint policies, run `python train_bestresponse.py ROLLOUT_DIR NUM_EPISODES_TO_TRAIN --ep-strs ep1 ep2 --agent-type all`. The `--ep-strs` argument specifies which episodes to run on (for example, policies from episode 0, 10000, and 200000). These must be episodes for which policies were saved. It is possible to specify a single agent type.
|
|
|
|
|
|
## What Will Be Saved?
|
|
|
|
A large amount of data will be saved -- one can set hyperparamter `train.save_dense_every` in the configuration dictionary (`hparams.yaml`/`constants.py`) to reduce this.
|
|
|
|
At the top level, an experiment directory stores the results of many runs in a hyperparameter sweep. Example structure:
|
|
|
|
```
|
|
experiment/experimentname/
|
|
rollout-999999-99999/
|
|
brconsumer/
|
|
...
|
|
brfirm/
|
|
episode_XXXX_consumer.npz
|
|
episode_XXXX_government.npz
|
|
episode_XXXX_firm.npz
|
|
saved_models/
|
|
consumer_policy_XXX.pt
|
|
firm_policy_XXX.pt
|
|
government_policy_XXX.pt.
|
|
brgovernment/
|
|
...
|
|
hparams.yaml
|
|
action_arrays.pickle
|
|
episode_XXXX_consumer.npz
|
|
episode_XXXX_government.npz
|
|
episode_XXXX_firm.npz
|
|
saved_models/
|
|
consumer_policy_XXX.pt
|
|
firm_policy_XXX.pt
|
|
government_policy_XXX.pt.
|
|
|
|
rollout-777777-77777/
|
|
...
|
|
```
|
|
|
|
Files:
|
|
|
|
`rollout-XXXXXX-XXX`: subdirectory containing all output for a single run.
|
|
|
|
`hparams.yaml`: configuration dictionary with hyperparameters
|
|
|
|
`action_arrays.pickle`: contains saved action arrays (allowing mapping action indices to the actual action, e.g. index 1 is price 1000.0, etc.)
|
|
|
|
`episode_XXXX_AGENTTYPE.npz`: Contains dense rollouts stored as the output of a numpy.savez call. When loaded, can be treated like a dictionary of numpy arrays. Has keys: `['states', 'actions', 'rewards', 'action_array', 'aux_array']` (view keys by using `.files`). `states`, `actions`, `rewards`, and `aux_array` all refer to saved copies of CUDA arrays (described below). `action_array` is a small array mapping action indices to the actual action.
|
|
|
|
`saved_models/AGENTTYPE_policy_XXX.pt`: a saved PyTorch state dict of the policy network, after episode XXX.
|
|
|
|
## Structure Of Arrays
|
|
|
|
`states` for any given agent type is an array storing observed states. It has shape `batch_size, ep_length, num_agents, agent_total_state_dim`.
|
|
|
|
`actions` is an array consisting of the action _indices_ (integers). For firms and government, it is of shape `batch_size, ep_length, num_agents`. For consumers, it is of shape `batch_size, ep_length, num_agents, num_action_heads`.
|
|
|
|
`rewards` stores total rewards, and is of shape `batch_size, ep_length, num_agents`.
|
|
|
|
The `aux_array` stores additional information and may differ per agent type. The consumer `aux_array` stores _actual_ consumption of each firm's good (as opposed to attempted consumption). The firm `aux_array` stores the amount bought by the export market.
|
|
|
|
## State Array Layout:
|
|
|
|
States observed by each agent consist of a global state, plus additional state dimensions per agent.
|
|
|
|
Global state: total dimension 4 * num_firms + 2 + 1
|
|
- prices: 1 per firm
|
|
- wages: 1 per firm
|
|
- inventories: 1 per firm
|
|
- overdemanded flag: 1 per firm
|
|
- time
|
|
|
|
Consumer additional state variables: total dimension global state + 2
|
|
- budget
|
|
- theta
|
|
|
|
Firm additional state variables: total dimension global state + 3 + num_firms
|
|
- budget
|
|
- capital
|
|
- production alpha
|
|
- one-hot representation identifying which firm
|
|
|
|
## What Gets Loaded And Written By BR Code?
|
|
|
|
The best response code loads in the `hparams.yaml` file, and the policies at a given time step (i.e. `saved_models/...policy_XXX.pt`). It then trains one of the policies while keeping the others fixed. Results are written to directories `brfirm`, `brconsumer`, `brgovernment` and contain dense rollouts and saved policy checkpoints, but from the best response training.
|
|
|
|
## Which Hyperparameters Are Managed And Where?
|
|
|
|
Initial values of state variables (budgets, initial wages, levels of capital, and so on) are set by the code in the method `__init_cuda_data_structs`. Some of these can be controlled from the hyperparameter dict; others are currently hardcoded.
|
|
|
|
Other hyperparameters are specified in the configuration dictionary.
|
|
|
|
Finally, the technology parameter (A) of the production function is currently hardcoded in the function call in `rbc/cuda/firm_rbc.cu`.
|