Dev Environment Setup
A MetricsDAO guide to setting up your data curation development environment.
MetricsDAO curators use three primary tools:
Tool | Used for |
|---|---|
Querying tables & connecting to third party tools | |
Viewing technical documentation, contributing & collaboration | |
Curating table designs |
See below for details on setup and access for each tool.
Once you have been given the temporary password associated with your username, be sure to log in to the Snowflake UI at https://app.snowflake.com/us-east-1/dqa62717/ using your temporary password. Once you’re logged in, you should automatically be asked by Snowflake to change your temporary password to a real password of your choosing. If not, you can always go to your profile page to update your password.
You have two options to contribute to MetricsDAO's Github repository:
- Option 1: Follow the Fork and Pull Workflow with your own Github (see below)
- Option 2: Be added as a contributor to the MetricsDAO Github and commit directly to project repositories with pull requests
The Fork and Pull Workflow is:
- Fork the MetricsDAO repository to your Github.
- Git clone from your forked repository to your local machine. ex:
git clonehttps://github.com/YourAccount/terra_dbt` - Create a branch for the changes you'd like to make. Example:
git branch readme-update. - Switch to the branch. Ex:
git checkout readme-update. - Make your changes on the branch and follow the rest of the steps in the Fork and Pull Workflow to notify the MetricsDAO repository owners to review your changes.
- Note: In fact, the fork-and-pull workflow is the only workflow supported if you’re using dbt Cloud for your dbt development environment (more on dbt below).
As of October 2022, the Data Curation pod accepts the following as development environments:
- dbt Cloud
- dbt on Linux/MacOS
All of these are just ways get a code development UI, be that the DBT Cloud IDE in the first case or VSCode in the latter two cases, to talk to dbt and then in turn getting dbt to talk to the MetricsDAO Snowflake data warehouse.
The quickest way to start working is to use dbt Cloud. The Data Curation pod has put together a video to get you familiar with dbt concepts using dbt Cloud. However, if you’re already familiar with dbt and just want to quickly setup dbt Cloud with the MetricsDAO Snowflake, skip to 49:25 of the video.
dbt is available for installation via several popular package managers, including Homebrew and pip.
Follow the instructions on dbt Labs' documentation to get dbt set-up on your machine:
To manage dependencies and account for differences in machines, Docker may be used to create a dbt terminal for development.
- Follow Docker setup
For Windows users, you'll need to install WSL and connect VSCode to WSL by
- Right-clicking VSCode and running VSCode as admin.
- Installing WSL by typing
wsl --installin VScode's terminal. - Following the rest of the VSCode WSL instructions to create a new WSL user.
- Installing the Remote Development extension (ms-vscode-remote.vscode-remote-extensionpack) in VSCode.
- Finally, restarting VSCode in a directory in which you'd like to work. For example,Finally, restarting VSCode in a directory in which you'd like to work. For example,
- `cd ~/metricsDAO/data_curation/terra_dbt`
- `code .`
- Clone the project or MetricsDao Github repo
git clone https://github.com/MetricsDAO/harmony_dbt - Copy
.env.sampleand create.env, replace the highlights below with your own account and passwords -
.png?alt=media)
- Install the dbt formatter from VS Code. For ease of use, turn on "format on save" by searching for "format" in the settings menu.
- henriblancke.vscode-dbt-formatter
- Assume you have completed all of the steps above successfully, you’re ready to start the dbt console where you can make and test your models.
- In VSCode’s terminal, type
cd <your dbt project>. Ex:cd terra_dbt. - Then run `make dbt-console`
- You can verify that the above command ran successfully by looking at the terminal prompt. It should have changed from your Linux bash promp to something like [email protected]://terra#. Alternatively, you can see in the Docker Desktop app that an instance of the terra_dbt is now running.
- C:\Users\xyz>make 'make' is not recognized as an internal or external command, operable program or batch file.
- Install make for windows @ http://gnuwin32.sourceforge.net/packages/make.htm
- Something doesn’t work on the docker container.
- Try git clean -xdf to clean up the whole repo.
- Your .env file will be removed and thus please remember to save your password.
- What Python version can I use?
- As of v1.0.0, dbt-core is compatible with Python versions 3.7, 3.8, and 3.9.
Last modified 1mo ago