The IamDataFrame class

class pyam.IamDataFrame(data, meta=None, index=['model', 'scenario'], **kwargs)

Scenario timeseries data and meta indicators

The class provides a number of diagnostic features (including validation of data, completeness of variables provided), processing tools (e.g., unit conversion), as well as visualization and plotting tools.

Parameters:

datapandas.DataFrame or file-like object as str or pathlib.Path

Scenario timeseries data following the IAMC data format or a supported variation as pandas object or a path to a file.

metapandas.DataFrame, optional

A dataframe with suitable ‘meta’ indicators for the new instance. The index will be downselected to scenarios present in data.

indexlist, optional

Columns to use for resulting IamDataFrame index.

kwargs

If value=<col>, melt column <col> to ‘value’ and use <col> name as ‘variable’; or mapping of required columns (IAMC_IDX) to any of the following:

• one column in data

• multiple columns, to be concatenated by |

• a string to be used as value for this column

Notes

A pandas.DataFrame can have the required dimensions as columns or index. R-style integer column headers (i.e., X2015) are acceptable.

When initializing an IamDataFrame from an xlsx file, pyam will per default parse all sheets starting with ‘data’ or ‘Data’ for timeseries and a sheet ‘meta’ to populate the respective table. Sheet names can be specified with kwargs sheet_name (‘data’) and meta_sheet_name (‘meta’), where values can be a string or a list and ‘*’ is interpreted as a wildcard. Calling the class with meta_sheet_name=False will skip the import of the ‘meta’ table.

When initializing an IamDataFrame from an object that is already an IamDataFrame instance, the new object will be hard-linked to all attributes of the original object - so any changes on one object (e.g., with inplace=True) may also modify the other object! This is intended behaviour and consistent with pandas but may be confusing for those who are not used to the pandas/Python universe.

Attributes:

compute

Access to advanced computation methods, see IamComputeAccessor

data

Return the timeseries data as a long pandas.DataFrame

dimensions

Return the list of data columns (index names & data coordinates)

empty

Indicator whether this object is empty

index

Return all model-scenario combinations as pandas.MultiIndex

model

Return the list of (unique) model names

region

Return the list of (unique) regions

scenario

Return the list of (unique) scenario names

time

The time index, i.e., axis labels related to the time domain.

time_domain

Indicator of the time domain: ‘year’, ‘datetime’, or ‘mixed’

unit

Return the list of (unique) units

unit_mapping

Return a dictionary of variables to (list of) corresponding units

variable

Return the list of (unique) variables

Methods

add(a, b, name[, axis, fillna, ...])	Add timeseries data items a and b along an axis
aggregate(variable[, components, method, ...])	Aggregate timeseries data by components or subcategories within each region.
aggregate_region(variable[, region, ...])	Aggregate timeseries data by subregions.
aggregate_time(variable[, column, value, ...])	Aggregate timeseries data by subannual time resolution
append(other[, ignore_meta_conflict, ...])	Append any IamDataFrame-like object to this object
apply(func, name[, axis, fillna, append, args])	Apply a function to components of timeseries data along an axis
as_pandas([meta_cols])	Return object as a pandas.DataFrame
categorize(name, value, criteria[, color, ...])	Assign scenarios to a category according to specific criteria
check_aggregate(variable[, components, ...])	Check whether timeseries data matches the aggregation by its components.
check_aggregate_region(variable[, region, ...])	Check whether timeseries data matches the aggregation across subregions.
check_internal_consistency([components])	Check whether a scenario ensemble is internally consistent
col_apply(col, func, *args, **kwargs)	Apply a function to a column of data or meta
compute_bias(name, method, axis)	DEPRECATED - please use IamDataFrame.compute.bias()
convert_unit(current, to[, factor, ...])	Convert all timeseries data having current units to new units.
copy()	Make a deepcopy of this object
diff(mapping[, periods, append])	Compute the difference of timeseries data along the time dimension
divide(a, b, name[, axis, fillna, ...])	Divide the timeseries data items a and b along an axis
downscale_region(variable[, region, ...])	Downscale timeseries data to a number of subregions
equals(other)	Test if two objects contain the same data and meta indicators
export_meta(excel_writer[, sheet_name])	Write the 'meta' indicators of this object to an Excel sheet
filter([keep, inplace])	Return a (copy of a) filtered (downselected) IamDataFrame
get_data_column(column)	Return a column from the timeseries data in long format
head(*args, **kwargs)	Identical to pandas.DataFrame.head() operating on data
info([n, meta_rows, memory_usage])	Print a summary of the object index dimensions and meta indicators
interpolate(time[, inplace])	Interpolate missing values in the timeseries data
load_meta(path[, sheet_name, ignore_conflict])	Load 'meta' indicators from file
map_regions(map_col[, agg, copy_col, fname, ...])	Plot regional data for a single model, scenario, variable, and year
multiply(a, b, name[, axis, fillna, ...])	Multiply timeseries data items a and b along an axis
normalize([inplace])	Normalize data to a specific data point
offset([padding, fill_value, inplace])	Compute new data which is offset from a specific data point
pivot_table(index, columns[, values, ...])	Returns a pivot table
rename([mapping, inplace, append, ...])	Rename any index dimension or data coordinate.
require_variable(variable[, unit, year, ...])	Check whether all scenarios have a required variable
reset_exclude()	Reset exclusion assignment for all scenarios to exclude: False
set_meta(meta[, name, index])	Add meta indicators as pandas.Series, list or value (int/float/str)
set_meta_from_data(name[, method, column])	Add meta indicators from downselected timeseries data of self
slice([keep])	Return a (filtered) slice object of the IamDataFrame timeseries data index
subtract(a, b, name[, axis, fillna, ...])	Compute the difference of timeseries data items a and b along an axis
swap_time_for_year([subannual, inplace])	Convert the time dimension to year (as integer).
swap_year_for_time([inplace])	Convert the year and subannual dimensions to time (as datetime).
tail(*args, **kwargs)	Identical to pandas.DataFrame.tail() operating on data
timeseries([iamc_index])	Returns data as pandas.DataFrame in wide format
to_csv(path[, iamc_index])	Write timeseries data of this object to a csv file
to_datapackage(path)	Write object to a frictionless Data Package
to_excel(excel_writer[, sheet_name, ...])	Write object to an Excel spreadsheet
validate([criteria, exclude_on_fail])	Validate scenarios using criteria on timeseries values

add(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)

Add timeseries data items a and b along an axis

This function computes a + b. If a or b are lists, the method applies pandas.groupby().sum() on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.

Parameters:

a, bstr, list of str or a number

Items to be used for the addition.

namestr

Name of the computed timeseries data on the axis.

axisstr, optional

Axis along which to compute.

fillnadict or scalar, optional

Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form {arg: default}.

ignore_unitsbool or str, optional

Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).

appendbool, optional

Whether to append aggregated timeseries data to this instance.

Returns:

IamDataFrame or None

Computed timeseries data or None if append=True.

See also

subtract, multiply, divide

apply

Apply a custom function on the timeseries data along any axis.

aggregate

Aggregate timeseries data along the variable hierarchy.

aggregate_region

Aggregate timeseries data along the region dimension.

Notes

This function uses the pint package and the iam-units registry (read the docs) to handle units. pyam will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units '{:~}'.format(u) (see here for more information).

As a result, the notation of returned units may differ from the input format. For example, the unit EJ/yr may be reformatted to EJ / a.

aggregate(variable, components=None, method='sum', recursive=False, append=False)

Aggregate timeseries data by components or subcategories within each region.

Parameters:

variablestr or list of str

Variable(s) for which the aggregate will be computed.

componentslist of str, optional

Components to be aggregate, defaults to all subcategories of variable.

methodfunc or str, optional

Aggregation method, e.g. numpy.mean, numpy.sum, ‘min’, ‘max’.

recursivebool or str, optional

Iterate recursively (bottom-up) over all subcategories of variable. If there are existing intermediate variables, it validates the aggregated value. If recursive=’skip-validate’, it skips the validation.

appendbool, optional

Whether to append aggregated timeseries data to this instance.

Returns:

IamDataFrame or None

Aggregated timeseries data or None if append=True.

See also

add

Add timeseries data items along an axis.

aggregate_region

Aggregate timeseries data along the region dimension.

Notes

The aggregation function interprets any missing values (numpy.nan) for individual components as 0.

aggregate_region(variable, region='World', subregions=None, components=False, method='sum', weight=None, append=False, drop_negative_weights=True)

Aggregate timeseries data by subregions.

This function allows to add variable sub-categories that are only defined at the region level by setting components=True

Parameters:

variablestr or list of str

Variable(s) to be aggregated.

regionstr, optional

Region to which data will be aggregated

subregionslist of str, optional

List of subregions, defaults to all regions other than region.

componentsbool or list of str, optional

Variables at the region level to be included in the aggregation (ignored if False); if True, use all sub-categories of variable included in region but not in any of the subregions; or explicit list of variables.

methodfunc or str, optional

Method to use for aggregation, e.g. numpy.mean, numpy.sum, ‘min’, ‘max’.

weightstr, optional

Variable to use as weight for the aggregation (currently only supported with method=’sum’).

appendbool, optional

Append the aggregate timeseries to self and return None, else return aggregate timeseries as new IamDataFrame.

drop_negative_weightsbool, optional

Removes any aggregated values that are computed using negative weights.

Returns:

IamDataFrame or None

Aggregated timeseries data or None if append=True.

See also

add

Add timeseries data items a and b along an axis

aggregate

Aggregate timeseries data along the variable hierarchy.

aggregate_time(variable, column='subannual', value='year', components=None, method='sum', append=False)

Aggregate timeseries data by subannual time resolution

Parameters:

variablestr or list of str

variable(s) to be aggregated

columnstr, optional

the data column to be used as subannual time representation

valuestr, optional

the name of the aggregated (subannual) time

componentslist of str

subannual timeslices to be aggregated; defaults to all subannual timeslices other than value

methodfunc or str, optional

method to use for aggregation, e.g. numpy.mean(), numpy.sum(), ‘min’, ‘max’

appendbool, optional

append the aggregate timeseries to self and return None, else return aggregate timeseries as new IamDataFrame

append(other, ignore_meta_conflict=False, inplace=False, verify_integrity=True, **kwargs)

Append any IamDataFrame-like object to this object

Indicators in other.meta that are not in self.meta are merged. Missing values are set to NaN. Conflicting data rows always raise a ValueError.

Parameters:

otherIamDataFrame, pandas.DataFrame or data file

Any object castable as IamDataFrame to be appended

ignore_meta_conflictbool, optional

If False and other is an IamDataFrame, raise an error if any meta columns present in self and other are not identical.

inplacebool, optional

If True, do operation inplace and return None

verify_integritybool, optional

If True, verify integrity of index

kwargs

Passed to IamDataFrame(other, **kwargs) if other is not already an IamDataFrame

Returns:

IamDataFrame

If inplace is False.

None

If inplace is True.

Raises:

ValueError

If time domain or other timeseries data index dimension don’t match.

apply(func, name, axis='variable', fillna=None, append=False, args=(), **kwds)

Apply a function to components of timeseries data along an axis

This function computes a function func using timeseries data selected along an axis downselected by keyword arguments. The length of components needs to match the number of required arguments of func.

Parameters:

funcfunction

Function to apply to components along axis.

namestr

Name of the computed timeseries data on the axis.

axisstr, optional

Axis along which to compute.

fillnadict or scalar, optional

Value to fill holes when rows are not defined for items in args or kwds. Can be a scalar or a dictionary of the form {kwd: default}.

appendbool, optional

Whether to append aggregated timeseries data to this instance.

argstuple or list of str

List of variables to pass as positional arguments to func.

**kwds

Additional keyword arguments to pass as keyword arguments to func. If the name of a variable is given, the associated timeseries is passed. Otherwise the value itself is passed.

Returns:

IamDataFrame or None

Computed timeseries data or None if append=True.

See also

add, subtract, multiply, divide, diff

Notes

This function uses the pint package and the iam-units registry (read the docs) to handle units. pyam uses abbreviated units '{:~}'.format(u) (see here for more information).

As a result, the notation of returned units may differ from the input format. For example, the unit EJ/yr may be reformatted to EJ / a.

as_pandas(meta_cols=True)

Return object as a pandas.DataFrame

Parameters:

meta_colslist, default None

join data with all meta columns if True (default) or only with columns in list, or return copy of data if False

categorize(name, value, criteria, color=None, marker=None, linestyle=None)

Assign scenarios to a category according to specific criteria

Parameters:

namestr

column name of the ‘meta’ table

valuestr

category identifier

criteriadict

dictionary with variables mapped to applicable checks (‘up’ and ‘lo’ for respective bounds, ‘year’ for years - optional)

colorstr, optional

assign a color to this category for plotting

markerstr, optional

assign a marker to this category for plotting

linestylestr, optional

assign a linestyle to this category for plotting

check_aggregate(variable, components=None, method='sum', exclude_on_fail=False, multiplier=1, **kwargs)

Check whether timeseries data matches the aggregation by its components.

Parameters:

variablestr or list of str

Variable(s) checked for matching aggregation of sub-categories.

componentslist of str, default None

List of variables to aggregate, defaults to sub-categories of variable.

methodfunc or str, optional

Method to use for aggregation, e.g. numpy.mean, :aby:`numpy.sum`, ‘min’, ‘max’.

exclude_on_failbool, optional

Flag scenarios failing validation as exclude: True.

multipliernumber, optional

Multiplicative factor when comparing variable and sum of components.

kwargsTolerance arguments for comparison of values

Passed to numpy.isclose().

Returns:

pandas.DataFrame or None

Data where variables and aggregate does not match.

check_aggregate_region(variable, region='World', subregions=None, components=False, method='sum', weight=None, exclude_on_fail=False, drop_negative_weights=True, **kwargs)

Check whether timeseries data matches the aggregation across subregions.

Parameters:

variablestr or list of str

Variable(s) to be checked for matching aggregation of subregions.

regionstr, optional

Region to be checked for matching aggregation of subregions.

subregionslist of str, optional

List of subregions, defaults to all regions other than region.

componentsbool or list of str, optional

Variables at the region level to be included in the aggregation (ignored if False); if True, use all sub-categories of variable included in region but not in any of the subregions; or explicit list of variables.

methodfunc or str, optional

Method to use for aggregation, e.g. numpy.mean, numpy.sum, ‘min’, ‘max’.

weightstr, optional

Variable to use as weight for the aggregation (currently only supported with method=’sum’).

exclude_on_failboolean, optional

Flag scenarios failing validation as exclude: True.

drop_negative_weightsbool, optional

Removes any aggregated values that are computed using negative weights

kwargsTolerance arguments for comparison of values

Passed to numpy.isclose().

Returns:

pandas.DataFrame or None

Data where variables and region-aggregate does not match.

check_internal_consistency(components=False, **kwargs)

Check whether a scenario ensemble is internally consistent

We check that all variables are equal to the sum of their sectoral components and that all the regions add up to the World total. If the check is passed, None is returned, otherwise a DataFrame of inconsistent variables is returned.

Note: at the moment, this method’s regional checking is limited to checking that all the regions sum to the World region. We cannot make this more automatic unless we store how the regions relate, see this issue.

Parameters:

kwargsarguments for comparison of values

passed to numpy.isclose()

componentsbool, optional

passed to check_aggregate_region() if True, use all sub-categories of each variable included in World but not in any of the subregions; if False, only aggregate variables over subregions

col_apply(col, func, *args, **kwargs)

Apply a function to a column of data or meta

Parameters:

col: str

column in either data or meta dataframes

func: function

function to apply

property compute

Access to advanced computation methods, see IamComputeAccessor

compute_bias(name, method, axis)

DEPRECATED - please use IamDataFrame.compute.bias()

convert_unit(current, to, factor=None, registry=None, context=None, inplace=False)

Convert all timeseries data having current units to new units.

If factor is given, existing values are multiplied by it, and the to units are assigned to the ‘unit’ column.

Otherwise, the pint package is used to convert from current -> to units without an explicit conversion factor. Pint natively handles conversion between any standard (SI) units that have compatible dimensionality, such as exajoule to terawatt-hours, EJ -> TWh, or tonne per year to gram per second, t / yr -> g / sec.

The default registry includes additional unit definitions relevant for integrated assessment models and energy systems analysis, via the iam-units package. This registry can also be accessed directly, using:

from iam_units import registry

When using this registry, current and to may contain the symbols of greenhouse gas (GHG) species, such as ‘CO2e’, ‘C’, ‘CH4’, ‘N2O’, ‘HFC236fa’, etc., as well as lower-case aliases like ‘co2’ supported by pyam. In this case, context must be the name of a specific global warming potential (GWP) metric supported by iam_units, e.g. ‘AR5GWP100’ (optionally prefixed by ‘gwp_’, e.g. ‘gwp_AR5GWP100’).

Rows with units other than current are not altered.

Parameters:

currentstr

Current units to be converted.

tostr

New unit (to be converted to) or symbol for target GHG species. If only the GHG species is provided, the units (e.g. Mt / year) will be the same as current, and an expression combining units and species (e.g. ‘Mt CO2e / yr’) will be placed in the ‘unit’ column.

factorvalue, optional

Explicit factor for conversion without pint.

registrypint.UnitRegistry, optional

Specific unit registry to use for conversion. Default: the iam-units registry.

contextstr or pint.Context, optional

(Name of) the context to use in conversion. Required when converting between GHG species using GWP metrics, unless the species indicated by current and to are the same.

inplacebool, optional

Whether to return a new IamDataFrame.

Returns:

IamDataFrame

If inplace is False.

None

If inplace is True.

Raises:

pint.UndefinedUnitError

if attempting a GWP conversion but context is not given.

pint.DimensionalityError

without factor, when current and to are not compatible units.

copy()

Make a deepcopy of this object

See copy.deepcopy() for details.

property data

Return the timeseries data as a long pandas.DataFrame

diff(mapping, periods=1, append=False)

Compute the difference of timeseries data along the time dimension

This methods behaves as if applying pandas.DataFrame.diff() on the timeseries data in wide format. By default, the diff-value in period t is computed as x[t] - x[t-1].

Parameters:

mappingdict

Mapping of variable item(s) to the name(s) of the diff-ed timeseries data, e.g.,

{"current variable": "name of diff-ed variable", ...}

periodsint, optional

Periods to shift for calculating difference, accepts negative values; passed to pandas.DataFrame.diff().

appendbool, optional

Whether to append computed timeseries data to this instance.

Returns:

IamDataFrame or None

Computed timeseries data or None if append=True.

See also

subtract, apply, interpolate

Notes

This method behaves as if applying pandas.DataFrame.diff() by row in a wide data format, so the difference is computed on the previous existing value. This can lead to unexpected results if the data has inconsistent period lengths.

Use the following to ensure that no missing values exist prior to computing the difference:

df.interpolate(time=df.year)

property dimensions

Return the list of data columns (index names & data coordinates)

divide(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)

Divide the timeseries data items a and b along an axis

This function computes a / b. If a or b are lists, the method applies pandas.groupby().sum() on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.

Parameters:

a, bstr, list of str or a number

Items to be used for the division.

namestr

Name of the computed timeseries data on the axis.

axisstr, optional

Axis along which to compute.

fillnadict or scalar, optional

Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form {arg: default}.

ignore_unitsbool or str, optional

Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).

appendbool, optional

Whether to append aggregated timeseries data to this instance.

Returns:

IamDataFrame or None

Computed timeseries data or None if append=True.

See also

add, subtract, multiply

apply

Apply a custom function on the timeseries data along any axis.

Notes

This function uses the pint package and the iam-units registry (read the docs) to handle units. pyam will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units '{:~}'.format(u) (see here for more information).

As a result, the notation of returned units may differ from the input format. For example, the unit EJ/yr may be reformatted to EJ / a.

downscale_region(variable, region='World', subregions=None, proxy=None, weight=None, append=False)

Downscale timeseries data to a number of subregions

Parameters:

variablestr or list of str

variable(s) to be downscaled

regionstr, optional

region from which data will be downscaled

subregionslist of str, optional

list of subregions, defaults to all regions other than region (if using proxy) or region index (if using weight)

proxystr, optional

variable (within the IamDataFrame) to be used as proxy for regional downscaling

weightclass:pandas.DataFrame, optional

dataframe with time dimension as columns (year or datetime.datetime) and regions[, model, scenario] as index

appendbool, optional

append the downscaled timeseries to self and return None, else return downscaled data as new IamDataFrame

property empty

Indicator whether this object is empty

equals(other)

Test if two objects contain the same data and meta indicators

This function allows two IamDataFrame instances to be compared against each other to see if they have the same timeseries data and meta indicators. nan’s in the same location of the meta table are considered equal.

Parameters:

otherIamDataFrame

the other IamDataFrame to be compared with self

export_meta(excel_writer, sheet_name='meta')

Write the ‘meta’ indicators of this object to an Excel sheet

Parameters:

excel_writerstr, path object or ExcelWriter object

any valid string path, pathlib.Path or pandas.ExcelWriter

sheet_namestr

name of sheet which will contain dataframe of ‘meta’ indicators

filter(keep=True, inplace=False, **kwargs)

Return a (copy of a) filtered (downselected) IamDataFrame

Parameters:

keepbool, optional

Keep all scenarios satisfying the filters (if True) or the inverse.

inplacebool, optional

If True, do operation inplace and return None.

**kwargs

Passed to slice().

get_data_column(column)

Return a column from the timeseries data in long format

Equivalent to IamDataFrame.data[column].

Parameters:

columnstr

The column name.

Returns:

pd.Series

head(*args, **kwargs)

Identical to pandas.DataFrame.head() operating on data

property index

Return all model-scenario combinations as pandas.MultiIndex

The index allows to loop over the available model-scenario combinations using:

for model, scenario in df.index:
    ...

info(n=80, meta_rows=5, memory_usage=False)

Print a summary of the object index dimensions and meta indicators

Parameters:

nint

The maximum line length

meta_rowsint

The maximum number of meta indicators printed

interpolate(time, inplace=False, **kwargs)

Interpolate missing values in the timeseries data

This method uses pandas.DataFrame.interpolate(), which applies linear interpolation by default

Parameters:

timeint or datetime, or list-like thereof

Year or datetime.datetime to be interpolated. This must match the datetime/year format of self.

inplacebool, optional

if True, do operation inplace and return None

kwargs

passed to pandas.DataFrame.interpolate()

load_meta(path, sheet_name='meta', ignore_conflict=False, *args, **kwargs)

Load ‘meta’ indicators from file

Parameters:

pathstr, pathlib.Path or pandas.ExcelFile

A valid path or instance of an xlsx or csv file

sheet_namestr, optional

Name of the sheet to be parsed (if xlsx)

ignore_conflictbool, optional

If True, values in path take precedence over existing meta. If False, raise an error in case of conflicts.

kwargs

Passed to pandas.read_excel() or pandas.read_csv()

map_regions(map_col, agg=None, copy_col=None, fname=None, region_col=None, remove_duplicates=False, inplace=False)

Plot regional data for a single model, scenario, variable, and year

see pyam.plotting.region_plot() for all available options

Parameters:

map_colstr

The column used to map new regions to. Common examples include iso and 5_region.

aggstr, optional

Perform a data aggregation. Options include: sum.

copy_colstr, optional

Copy the existing region data into a new column for later use.

fnamestr, optional

Use a non-default region mapping file

region_colstring, optional

Use a non-default column name for regions to map from.

remove_duplicatesbool, optional

If there are duplicates in the mapping from one regional level to another, then remove these duplicates by counting the most common mapped value. This option is most useful when mapping from high resolution (e.g., model regions) to low resolution (e.g., 5_region).

inplacebool, optional

if True, do operation inplace and return None

property model

Return the list of (unique) model names

multiply(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)

Multiply timeseries data items a and b along an axis

This function computes a * b. If a or b are lists, the method applies pandas.groupby().sum() on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.

Parameters:

a, bstr, list of str or a number

Items to be used for the division.

namestr

Name of the computed timeseries data on the axis.

axisstr, optional

Axis along which to compute.

fillnadict or scalar, optional

Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form {arg: default}.

ignore_unitsbool or str, optional

Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).

appendbool, optional

Whether to append aggregated timeseries data to this instance.

Returns:

IamDataFrame or None

Computed timeseries data or None if append=True.

See also

add, subtract, divide

apply

Apply a custom function on the timeseries data along any axis.

Notes

This function uses the pint package and the iam-units registry (read the docs) to handle units. pyam will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units '{:~}'.format(u) (see here for more information).

As a result, the notation of returned units may differ from the input format. For example, the unit EJ/yr may be reformatted to EJ / a.

normalize(inplace=False, **kwargs)

Normalize data to a specific data point

Note: Currently only supports normalizing to a specific time.

Parameters:

inplacebool, optional

if True, do operation inplace and return None

kwargs

the column and value on which to normalize (e.g., year=2005)

offset(padding=0, fill_value=None, inplace=False, **kwargs)

Compute new data which is offset from a specific data point

For example, offsetting from year=2005 will provide data relative to year=2005 such that the value in 2005 is 0 and all other values value[year] - value[2005].

Conceptually this operation performs as: ` df - df.filter(**kwargs) + padding `

Note: Currently only supports normalizing to a specific time.

Parameters:

paddingfloat, optional

an additional offset padding

fill_valuefloat or None, optional

Applied on subtraction. Fills exisiting missing (NaN) values. See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.subtract.html

inplacebool, optional

if True, do operation inplace and return None

kwargs

the column and value on which to offset (e.g., year=2005)

pivot_table(index, columns, values='value', aggfunc='count', fill_value=None, style=None)

Returns a pivot table

Parameters:

indexstr or list of str

rows for Pivot table

columnsstr or list of str

columns for Pivot table

valuesstr, default ‘value’

dataframe column to aggregate or count

aggfuncstr or function, default ‘count’

function used for aggregation, accepts ‘count’, ‘mean’, and ‘sum’

fill_valuescalar, default None

value to replace missing values with

stylestr, default None

output style for pivot table formatting accepts ‘highlight_not_max’, ‘heatmap’

property region

Return the list of (unique) regions

rename(mapping=None, inplace=False, append=False, check_duplicates=True, **kwargs)

Rename any index dimension or data coordinate.

When renaming models or scenarios, the uniqueness of the index must be maintained, and the function will raise an error otherwise.

Renaming is only applied to any data row that matches for all columns given in mapping. Renaming can only be applied to the model and scenario columns, or to other data coordinates simultaneously.

Parameters:

mappingdict or kwargs

mapping of column name to rename-dictionary of that column

dict(<column_name>: {<current_name_1>: <target_name_1>,
                     <current_name_2>: <target_name_2>})

or kwargs as column_name={<current_name_1>: <target_name_1>, …}

inplacebool, optional

Do operation inplace and return None.

appendbool, optional

Whether to append aggregated timeseries data to this instance (if inplace=True) or to a returned new instance (if inplace=False).

check_duplicatesbool, optional

Check whether conflicts exist after renaming of timeseries data coordinates. If True, raise a ValueError; if False, rename and merge with groupby().sum().

Returns:

IamDataFrame or None

Aggregated timeseries data as new object or None if inplace=True.

require_variable(variable, unit=None, year=None, exclude_on_fail=False)

Check whether all scenarios have a required variable

Parameters:

variablestr

required variable

unitstr, default None

name of unit (optional)

yearint or list, default None

check whether the variable exists for ANY of the years (if a list)

exclude_on_failbool, default False

flag scenarios missing the required variables as exclude: True

reset_exclude()

Reset exclusion assignment for all scenarios to exclude: False

property scenario

Return the list of (unique) scenario names

set_meta(meta, name=None, index=None)

Add meta indicators as pandas.Series, list or value (int/float/str)

Parameters:

metapandas.DataFrame, pandas.Series, list, int, float or str

column to be added to ‘meta’ (by [‘model’, ‘scenario’] index if possible)

namestr, optional

meta column name (defaults to meta pandas.Series.name); either meta.name or the name kwarg must be defined

indexIamDataFrame, pandas.DataFrame or pandas.MultiIndex, optional

index to be used for setting meta column ([‘model’, ‘scenario’])

set_meta_from_data(name, method=None, column='value', **kwargs)

Add meta indicators from downselected timeseries data of self

Parameters:

namestr

column name of the ‘meta’ table

methodfunction, optional

method for aggregation (e.g., numpy.max); required if downselected data do not yield unique values

columnstr, optional

the column from data to be used to derive the indicator

kwargs

passed to filter() for downselected data

slice(keep=True, **kwargs)

Return a (filtered) slice object of the IamDataFrame timeseries data index

Parameters:

keepbool, optional

Keep all scenarios satisfying the filters (if True) or the inverse.

**kwargs

Arguments for filtering. See the “Notes”.

Returns:

IamSlice

Notes

The following arguments are available for filtering:

• ‘meta’ columns: filter by string value of that column

• ‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’: string or list of strings, where * can be used as a wildcard

• ‘index’: list of model, scenario 2-tuples or pandas.MultiIndex

• ‘level’: the “depth” of entries in the variable column (number of ‘|’) (excluding the strings given in the ‘variable’ argument)

• ‘year’: takes an integer (int/np.int64), a list of integers or a range. Note that the last year of a range is not included, so range(2010, 2015) is interpreted as [2010, …, 2014]

• ‘time_domain’: can be “year” or “datetime”

• arguments for filtering by datetime.datetime or np.datetime64 (‘month’, ‘hour’, ‘time’)

• ‘regexp=True’ disables pseudo-regexp syntax in pattern_match()

subtract(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)

Compute the difference of timeseries data items a and b along an axis

This function computes a - b. If a or b are lists, the method applies pandas.groupby().sum() on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.

Parameters:

a, bstr, list of str or a number

Items to be used for the subtraction.

namestr

Name of the computed timeseries data on the axis.

axisstr, optional

Axis along which to compute.

fillnadict or scalar, optional

Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form {arg: default}.

ignore_unitsbool or str, optional

Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).

appendbool, optional

Whether to append aggregated timeseries data to this instance.

Returns:

IamDataFrame or None

Computed timeseries data or None if append=True.

See also

add, multiply, divide

diff

Compute the difference of timeseries data along the time dimension.

apply

Apply a custom function on the timeseries data along any axis.

Notes

This function uses the pint package and the iam-units registry (read the docs) to handle units. pyam will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units '{:~}'.format(u) (see here for more information).

As a result, the notation of returned units may differ from the input format. For example, the unit EJ/yr may be reformatted to EJ / a.

swap_time_for_year(subannual=False, inplace=False)

Convert the time dimension to year (as integer).

Parameters:

subannualbool, str or func, optional

Merge non-year components of the “time” domain as new column “subannual”. Apply strftime() on the values of the “time” domain using subannual (if a string) or “%m-%d %H:%M%z” (if True). If it is a function, apply the function on the values of the “time” domain.

inplacebool, optional

If True, do operation inplace and return None.

Returns:

IamDataFrame or None

Object with altered time domain or None if inplace=True.

Raises:

ValueError

“time” is not a column of self.data

See also

swap_year_for_time

swap_year_for_time(inplace=False)

Convert the year and subannual dimensions to time (as datetime).

The method applies dateutil.parser.parse() on the combined columns year and subannual:

dateutil.parser.parse([f"{y}-{s}" for y, s in zip(year, subannual)])

Parameters:

inplacebool, optional

If True, do operation inplace and return None.

Returns:

IamDataFrame or None

Object with altered time domain or None if inplace=True.

Raises:

ValueError

“year” or “subannual” are not a column of self.data

See also

swap_time_for_year

tail(*args, **kwargs)

Identical to pandas.DataFrame.tail() operating on data

property time

The time index, i.e., axis labels related to the time domain.

Returns:

- Aclass:pandas.Int64Index if the time_domain is ‘year’

- Aclass:pandas.DatetimeIndex if the time_domain is ‘datetime’

- Aclass:pandas.Index if the time_domain is ‘mixed’

property time_domain

Indicator of the time domain: ‘year’, ‘datetime’, or ‘mixed’

timeseries(iamc_index=False)

Returns data as pandas.DataFrame in wide format

Parameters:

iamc_indexbool, optional

if True, use [‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’]; else, use all ‘data’ columns

Raises:

ValueError

IamDataFrame is empty

ValueError

reducing to IAMC-index yields an index with duplicates

to_csv(path, iamc_index=False, **kwargs)

Write timeseries data of this object to a csv file

Parameters:

pathstr or path object

file path or pathlib.Path

iamc_indexbool, default False

if True, use [‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’]; else, use all ‘data’ columns

to_datapackage(path)

Write object to a frictionless Data Package

More information: https://frictionlessdata.io

Returns the saved datapackage.Package (read the docs). When adding metadata (descriptors), please follow the template defined by https://github.com/OpenEnergyPlatform/metadata

Parameters:

pathstring or path object

any valid string path or pathlib.Path

to_excel(excel_writer, sheet_name='data', iamc_index=False, include_meta=True, **kwargs)

Write object to an Excel spreadsheet

Parameters:

excel_writerstr, path object or ExcelWriter object

any valid string path, pathlib.Path or pandas.ExcelWriter

sheet_namestring

name of sheet which will contain timeseries() data

iamc_indexbool, default False

if True, use [‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’]; else, use all ‘data’ columns

include_metaboolean or string

if True, write ‘meta’ to an Excel sheet name ‘meta’ (default); if this is a string, use it as sheet name

property unit

Return the list of (unique) units

property unit_mapping

Return a dictionary of variables to (list of) corresponding units

validate(criteria={}, exclude_on_fail=False)

Validate scenarios using criteria on timeseries values

Returns all scenarios which do not match the criteria and prints a log message, or returns None if all scenarios match the criteria.

When called with exclude_on_fail=True, scenarios not satisfying the criteria will be marked as exclude=True.

Parameters:

criteriadict

dictionary with variable keys and validation mappings

(‘up’ and ‘lo’ for respective bounds, ‘year’ for years)

exclude_on_failbool, optional

flag scenarios failing validation as exclude: True

Returns:

pandas.DataFrame

All data points that do not satisfy the criteria.

None

If all scenarios satisfy the criteria.

property variable

Return the list of (unique) variables