{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Query data from the IIASA database infrastructure\n",
    "\n",
    "The IIASA *Energy, Climate, and Environment* Program hosts a suite of **Scenario Explorer** instances and related database infrastructure to support analysis of integrated-assessment pathways in IPCC reports and model comparison projects. \n",
    "High-profile use cases include the [AR6 Scenario Explorer hosted by IIASA](https://data.ece.iiasa.ac.at/ar6) supporting the *IPCC' Sixth Assessment Report* (AR6) and the Horizon 2020 project [ENGAGE](https://data.ece.iiasa.ac.at/engage).\n",
    "\n",
    "IIASA's [modeling platform infrastructure](http://docs.ece.iiasa.ac.at/) and the scenario apps are not only a great resource on its own, but it also allows the underlying datasets to be queried directly via a Rest API.\n",
    "The **pyam** package takes advantage of this ability to allow you to easily pull data and work with it in your Python data processing and analysis workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access and permission management for project-internal databases\n",
    "\n",
    "By default, your can connect to all public scenario database instances.\n",
    "\n",
    "If you have credentials to connect to a private, project-internal database instance,  \n",
    "you can store this information by running the following command in a console:\n",
    "\n",
    "```\n",
    "ixmp4 login <username>\n",
    "\n",
    "```\n",
    "\n",
    "You will be prompted to enter your password.\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "Your username and password will be saved locally in plain-text for future use!\n",
    "\n",
    "</div>\n",
    "\n",
    "When connecting to a database, **pyam** will automatically search for the configuration in a known location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting to ixmp4 database instances hosted by IIASA\n",
    "\n",
    "The *Scenario Apps* use the **ixmp4** package as a database backend.\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "The *Scenario Services* team is currently migrating database instances from the legacy *Scenario Explorer* infrastructure  \n",
    "to the new *Scenario Apps* services based on the **ixmp4** package.\n",
    "\n",
    "</div>\n",
    "\n",
    "You can list all available ixmp4 platforms hosted by IIASA using the following function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyam.iiasa.platforms()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting to a (legacy) Scenario Explorer database\n",
    "\n",
    "Accessing the database connected to a **Scenario Explorer** is done via a **Connection** object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn = pyam.iiasa.Connection()\n",
    "conn.valid_connections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading data from a database hosted by IIASA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **pyam** package can read both from **ixmp4** databases (connected to a *Scenario App*)  \n",
    "and the (legacy) **Scenario Explorer** infrastructure with the same function.\n",
    "\n",
    "In this example, we will be retrieving data from the *IAMC 1.5°C Scenario Explorer hosted by IIASA*\n",
    "([link](https://data.ece.iiasa.ac.at/iamc-1.5c-explorer)),\n",
    "which provides the quantitative scenario ensemble underpinning\n",
    "the *IPCC Special Report on Global Warming of 1.5C* (SR15).\n",
    "\n",
    "This can be done either via the constructor:\n",
    "\n",
    "```\n",
    "pyam.iiasa.Connection('iamc15')\n",
    "```\n",
    "\n",
    "or, if you want to query multiple databases, via the explicit `connect()` method:\n",
    "\n",
    "```\n",
    "conn = pyam.iiasa.Connection()\n",
    "conn.connect('iamc15')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also provide some convenience functions to shorten the amount of code you have to write. Under the hood, `read_iiasa()` is just opening a connection to a database API and sends a query to the resource.\n",
    "\n",
    "In this tutorial, we will query specific subsets of data in a manner similar to `pyam.IamDataFrame.filter()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pyam.read_iiasa(\n",
    "    \"iamc15\",\n",
    "    model=\"MESSAGEix*\",\n",
    "    variable=[\"Emissions|CO2\", \"Primary Energy|Coal\"],\n",
    "    region=\"World\",\n",
    "    meta=[\"category\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we pulled out all times series data for model(s) that start with 'MESSAGEix' that are in the 'World' region and associated with the two named variables. We also added the meta column \"category\", which tells us the climate impact categorisation of each scenario as assessed in the IPCC SR15.\n",
    "\n",
    "Let's plot CO2 emissions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.filter(variable=\"Emissions|CO2\").plot(\n",
    "    color=\"category\", legend=dict(loc=\"center left\", bbox_to_anchor=(1.0, 0.5))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now continue doing all of your analysis!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.plot.scatter(\n",
    "    x=\"Primary Energy|Coal\",\n",
    "    y=\"Emissions|CO2\",\n",
    "    color=\"category\",\n",
    "    legend=dict(loc=\"center left\", bbox_to_anchor=(1.0, 0.5)),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data resource\n",
    "\n",
    "If you're interested in what data is available in the data source, you can use **pyam.iiasa.Connection** to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn = pyam.iiasa.Connection(\"iamc15\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Connection** object has a number of useful functions for listing what's available in the data resource.\n",
    "These functions follow the conventions of the **IamDataFrame** class (where possible).\n",
    "\n",
    "A few of them are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.models().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.scenarios().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.variables().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.regions().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A number of different categorization and quantitative indicators are available for model/scenario combinations.  \n",
    "These are usually called 'meta' indicators in **pyam**.\n",
    "\n",
    "We queried the meta-indicator \"category\" in the above example, but there are many more.\n",
    "You can get a list with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.meta_columns.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can directly query the **Connection**, which will return a **pyam.IamDataFrame**..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = conn.query(\n",
    "    model=\"MESSAGEix*\",\n",
    "    variable=[\"Emissions|CO2\", \"Primary Energy|Coal\"],\n",
    "    region=\"World\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...so that you can directly continue with your analysis and visualization workflow using **pyam**!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.filter(variable=\"Primary Energy|Coal\").plot(\n",
    "    color=\"scenario\", legend=dict(loc=\"center left\", bbox_to_anchor=(1.0, 0.5))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading all of a large database may take a few minutes on some connections. To save time when writing code you may reuse, you can save a local version of the database via the __lazy_read_iiasa__ function. This is given a file location as well as whatever connection options we saw above. The first time the code is run, the result is stored there, and the code will read it from there on subsequent attempts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lazy_df = pyam.lazy_read_iiasa(\n",
    "    file=\"./tmp/messageix_co2_coal_data.csv\",\n",
    "    name=\"iamc15\",\n",
    "    model=\"MESSAGEix*\",\n",
    "    variable=[\"Emissions|CO2\", \"Primary Energy|Coal\"],\n",
    "    region=\"World\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}