The IamDataFrame class¶
- class pyam.IamDataFrame(data, meta=None, index=['model', 'scenario'], **kwargs)[source]¶
Scenario timeseries data and meta indicators
The class provides a number of diagnostic features (including validation of data, completeness of variables provided), processing tools (e.g., unit conversion), as well as visualization and plotting tools.
- Parameters
- data
pandas.DataFrame
,ixmp.Scenario
, or file-like object as str orpathlib.Path
Scenario timeseries data following the IAMC data format or a supported variation as pandas object, a path to a file, or a scenario of an ixmp instance.
- meta
pandas.DataFrame
, optional A dataframe with suitable ‘meta’ indicators for the new instance. The index will be downselected to scenarios present in data.
- indexlist, optional
Columns to use for resulting IamDataFrame index.
- kwargs
If value=<col>, melt column <col> to ‘value’ and use <col> name as ‘variable’; or mapping of required columns (
IAMC_IDX
) to any of the following:one column in data
multiple columns, to be concatenated by
|
a string to be used as value for this column
- data
Notes
A
pandas.DataFrame
can have the required dimensions as columns or index. R-style integer column headers (i.e., X2015) are acceptable.When initializing an
IamDataFrame
from an xlsx file,pyam
will per default parse all sheets starting with ‘data’ for timeseries and a sheet ‘meta’ to populate the respective table. Sheet names can be specified with kwargssheet_name
(‘data’) andmeta_sheet_name
(‘meta’), where values can be a string or a list and ‘*’ is interpreted as a wildcard. Calling the class withmeta_sheet_name=False
will skip the import of the ‘meta’ table.When initializing an
IamDataFrame
from an object that is already anIamDataFrame
instance, the new object will be hard-linked to all attributes of the original object - so any changes on one object (e.g., withinplace=True
) may also modify the other object! This is intended behaviour and consistent with pandas but may be confusing for those who are not used to the pandas/Python universe.- Attributes
data
Return the timeseries data as a long
pandas.DataFrame
dimensions
Return the list of data columns (index names & data coordinates)
empty
Indicator whether this object is empty
index
Return all model-scenario combinations as
pandas.MultiIndex
model
Return the list of (unique) model names
region
Return the list of (unique) regions
scenario
Return the list of (unique) scenario names
unit
Return the list of (unique) units
unit_mapping
Return a dictionary of variables to (list of) correspoding units
variable
Return the list of (unique) variables
Methods
add
(a, b, name[, axis, fillna, ...])Add timeseries data items a and b along an axis
aggregate
(variable[, components, method, ...])Aggregate timeseries by components or subcategories within each region
aggregate_region
(variable[, region, ...])Aggregate a timeseries over a number of subregions
aggregate_time
(variable[, column, value, ...])Aggregate a timeseries over a subannual time resolution
append
(other[, ignore_meta_conflict, ...])Append any IamDataFrame-like object to this object
apply
(func, name[, axis, fillna, append, args])Apply a function to components of timeseries data along an axis
as_pandas
([meta_cols])Return object as a pandas.DataFrame
categorize
(name, value, criteria[, color, ...])Assign scenarios to a category according to specific criteria
check_aggregate
(variable[, components, ...])Check whether a timeseries matches the aggregation of its components
check_aggregate_region
(variable[, region, ...])Check whether a timeseries matches the aggregation across subregions
check_internal_consistency
([components])Check whether a scenario ensemble is internally consistent
col_apply
(col, func, *args, **kwargs)Apply a function to a column of data or meta
compute_bias
(name, method, axis)Compute the bias weights and add to 'meta'
convert_unit
(current, to[, factor, ...])Convert all timeseries data having current units to new units.
copy
()Make a deepcopy of this object
divide
(a, b, name[, axis, fillna, ...])Divide the timeseries data items a and b along an axis
downscale_region
(variable[, region, ...])Downscale a timeseries to a number of subregions
equals
(other)Test if two objects contain the same data and meta indicators
export_meta
(excel_writer[, sheet_name])Write the 'meta' indicators of this object to an Excel sheet
filter
([keep, inplace])Return a (copy of a) filtered (downselected) IamDataFrame
get_data_column
(column)Return a column from the timeseries data in long format
head
(*args, **kwargs)Identical to
pandas.DataFrame.head()
operating on datainfo
([n, meta_rows, memory_usage])Print a summary of the object index dimensions and meta indicators
interpolate
(time[, inplace])Interpolate missing values in the timeseries data
load_meta
(path[, sheet_name, ignore_conflict])Load 'meta' indicators from file
map_regions
(map_col[, agg, copy_col, fname, ...])Plot regional data for a single model, scenario, variable, and year
multiply
(a, b, name[, axis, fillna, ...])Multiply timeseries data items a and b along an axis
normalize
([inplace])Normalize data to a specific data point
pivot_table
(index, columns[, values, ...])Returns a pivot table
rename
([mapping, inplace, append, ...])Rename any index dimension or data coordinate.
require_variable
(variable[, unit, year, ...])Check whether all scenarios have a required variable
Reset exclusion assignment for all scenarios to exclude: False
set_meta
(meta[, name, index])Add meta indicators as pandas.Series, list or value (int/float/str)
set_meta_from_data
(name[, method, column])Add meta indicators from downselected timeseries data of self
subtract
(a, b, name[, axis, fillna, ...])Compute the difference of timeseries data items a and b along an axis
swap_time_for_year
([subannual, inplace])Convert the time dimension to year (as integer).
swap_year_for_time
([inplace])Convert the year and subannual dimensions to time (as datetime).
tail
(*args, **kwargs)Identical to
pandas.DataFrame.tail()
operating on datatimeseries
([iamc_index])Returns data as
pandas.DataFrame
in wide formatto_csv
(path[, iamc_index])Write timeseries data of this object to a csv file
to_datapackage
(path)Write object to a frictionless Data Package
to_excel
(excel_writer[, sheet_name, ...])Write object to an Excel spreadsheet
validate
([criteria, exclude_on_fail])Validate scenarios using criteria on timeseries values
- add(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)[source]¶
Add timeseries data items a and b along an axis
This function computes a + b. If a or b are lists, the method applies
pandas.groupby().sum()
on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.- Parameters
- a, bstr, list of str or a number
Items to be used for the addition.
- namestr
Name of the computed timeseries data on the axis.
- axisstr, optional
Axis along which to compute.
- fillnadict or scalar, optional
Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form
{arg: default}
.- ignore_unitsbool or str, optional
Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).
- appendbool, optional
Whether to append aggregated timeseries data to this instance.
- Returns
IamDataFrame
or NoneComputed timeseries data or None if append=True.
See also
Notes
This function uses the
pint
package and theiam-units
registry (read the docs) to handle units.pyam
will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units'{:~}'.format(u)
(see here for more information).As a result, the notation of returned units may differ from the input format. For example, the unit
EJ/yr
may be reformatted toEJ / a
.
- aggregate(variable, components=None, method='sum', recursive=False, append=False)[source]¶
Aggregate timeseries by components or subcategories within each region
- Parameters
- variablestr or list of str
Variable(s) for which the aggregate will be computed.
- componentslist of str, optional
Components to be aggregate, defaults to all subcategories of variable.
- methodfunc or str, optional
Aggregation method, e.g.
numpy.mean()
,numpy.sum()
, ‘min’, ‘max’- recursivebool or str, optional
Iterate recursively (bottom-up) over all subcategories of variable. If there are existing intermediate variables, it validates the aggregated value. If recursive=’skip-validate’, it skips the validation.
- appendbool, optional
Whether to append aggregated timeseries data to this instance.
- Returns
IamDataFrame
or NoneAggregated timeseries data or None if append=True.
See also
add
Add timeseries data items along an axis.
aggregate_region
Aggregate timeseries data along the region dimension.
Notes
The aggregation function interprets any missing values (
numpy.nan
) for individual components as 0.
- aggregate_region(variable, region='World', subregions=None, components=False, method='sum', weight=None, append=False, drop_negative_weights=True)[source]¶
Aggregate a timeseries over a number of subregions
This function allows to add variable sub-categories that are only defined at the region level by setting components=True
- Parameters
- variablestr or list of str
variable(s) to be aggregated
- regionstr, optional
region to which data will be aggregated
- subregionslist of str, optional
list of subregions, defaults to all regions other than region
- componentsbool or list of str, optional
variables at the region level to be included in the aggregation (ignored if False); if True, use all sub-categories of variable included in region but not in any of the subregions; or explicit list of variables
- methodfunc or str, optional
method to use for aggregation, e.g.
numpy.mean()
,numpy.sum()
, ‘min’, ‘max’- weightstr, optional
variable to use as weight for the aggregation (currently only supported with method=’sum’)
- appendbool, optional
append the aggregate timeseries to self and return None, else return aggregate timeseries as new
IamDataFrame
- drop_negative_weightsbool, optional
removes any aggregated values that are computed using negative weights
- Returns
IamDataFrame
or NoneAggregated timeseries data or None if append=True.
- aggregate_time(variable, column='subannual', value='year', components=None, method='sum', append=False)[source]¶
Aggregate a timeseries over a subannual time resolution
- Parameters
- variablestr or list of str
variable(s) to be aggregated
- columnstr, optional
the data column to be used as subannual time representation
- valuestr, optional
the name of the aggregated (subannual) time
- componentslist of str
subannual timeslices to be aggregated; defaults to all subannual timeslices other than value
- methodfunc or str, optional
method to use for aggregation, e.g.
numpy.mean()
,numpy.sum()
, ‘min’, ‘max’- appendbool, optional
append the aggregate timeseries to self and return None, else return aggregate timeseries as new
IamDataFrame
- append(other, ignore_meta_conflict=False, inplace=False, verify_integrity=True, **kwargs)[source]¶
Append any IamDataFrame-like object to this object
Indicators in other.meta that are not in self.meta are merged. Missing values are set to NaN. Conflicting data rows always raise a ValueError.
- Parameters
- otherIamDataFrame, ixmp.Scenario, pandas.DataFrame or data file
Any object castable as IamDataFrame to be appended
- ignore_meta_conflictbool, optional
If False and other is an IamDataFrame, raise an error if any meta columns present in self and other are not identical.
- inplacebool, optional
If True, do operation inplace and return None
- verify_integritybool, optional
If True, verify integrity of index
- kwargs
Passed to
IamDataFrame(other, **kwargs)
if other is not already an IamDataFrame
- Returns
- Raises
- ValueError
If time domain or other timeseries data index dimension don’t match.
- apply(func, name, axis='variable', fillna=None, append=False, args=(), **kwds)[source]¶
Apply a function to components of timeseries data along an axis
This function computes a function func using timeseries data selected along an axis downselected by keyword arguments. The length of components needs to match the number of required arguments of func.
- Parameters
- funcfunction
Function to apply to components along axis.
- namestr
Name of the computed timeseries data on the axis.
- axisstr, optional
Axis along which to compute.
- fillnadict or scalar, optional
Value to fill holes when rows are not defined for items in args or kwds. Can be a scalar or a dictionary of the form
{kwd: default}
.- appendbool, optional
Whether to append aggregated timeseries data to this instance.
- argstuple or list of str
List of variables to pass as positional arguments to func.
- **kwds
Additional keyword arguments to pass as keyword arguments to func. If the name of a variable is given, the associated timeseries is passed. Otherwise the value itself is passed.
- Returns
IamDataFrame
or NoneComputed timeseries data or None if append=True.
Notes
This function uses the
pint
package and theiam-units
registry (read the docs) to handle units.pyam
uses abbreviated units'{:~}'.format(u)
(see here for more information).As a result, the notation of returned units may differ from the input format. For example, the unit
EJ/yr
may be reformatted toEJ / a
.
- as_pandas(meta_cols=True)[source]¶
Return object as a pandas.DataFrame
- Parameters
- meta_colslist, default None
join data with all meta columns if True (default) or only with columns in list, or return copy of data if False
- categorize(name, value, criteria, color=None, marker=None, linestyle=None)[source]¶
Assign scenarios to a category according to specific criteria
- Parameters
- namestr
column name of the ‘meta’ table
- valuestr
category identifier
- criteriadict
dictionary with variables mapped to applicable checks (‘up’ and ‘lo’ for respective bounds, ‘year’ for years - optional)
- colorstr, optional
assign a color to this category for plotting
- markerstr, optional
assign a marker to this category for plotting
- linestylestr, optional
assign a linestyle to this category for plotting
- check_aggregate(variable, components=None, method='sum', exclude_on_fail=False, multiplier=1, **kwargs)[source]¶
Check whether a timeseries matches the aggregation of its components
- Parameters
- variablestr or list of str
variable(s) checked for matching aggregation of sub-categories
- componentslist of str, default None
list of variables, defaults to all sub-categories of variable
- methodfunc or str, optional
method to use for aggregation, e.g.
numpy.mean()
,numpy.sum()
, ‘min’, ‘max’- exclude_on_failbool, optional
flag scenarios failing validation as exclude: True
- multipliernumber, optional
factor when comparing variable and sum of components
- kwargsarguments for comparison of values
passed to
numpy.isclose()
- check_aggregate_region(variable, region='World', subregions=None, components=False, method='sum', weight=None, exclude_on_fail=False, drop_negative_weights=True, **kwargs)[source]¶
Check whether a timeseries matches the aggregation across subregions
- Parameters
- variablestr or list of str
variable(s) to be checked for matching aggregation of subregions
- regionstr, optional
region to be checked for matching aggregation of subregions
- subregionslist of str, optional
list of subregions, defaults to all regions other than region
- componentsbool or list of str, optional
variables at the region level to be included in the aggregation (ignored if False); if True, use all sub-categories of variable included in region but not in any of the subregions; or explicit list of variables
- methodfunc or str, optional
method to use for aggregation, e.g.
numpy.mean()
,numpy.sum()
, ‘min’, ‘max’- weightstr, optional
variable to use as weight for the aggregation (currently only supported with method=’sum’)
- exclude_on_failboolean, optional
flag scenarios failing validation as exclude: True
- drop_negative_weightsbool, optional
removes any aggregated values that are computed using negative weights
- kwargsarguments for comparison of values
passed to
numpy.isclose()
- check_internal_consistency(components=False, **kwargs)[source]¶
Check whether a scenario ensemble is internally consistent
We check that all variables are equal to the sum of their sectoral components and that all the regions add up to the World total. If the check is passed, None is returned, otherwise a DataFrame of inconsistent variables is returned.
Note: at the moment, this method’s regional checking is limited to checking that all the regions sum to the World region. We cannot make this more automatic unless we store how the regions relate, see this issue.
- Parameters
- kwargsarguments for comparison of values
passed to
numpy.isclose()
- componentsbool, optional
passed to
check_aggregate_region()
if True, use all sub-categories of each variable included in World but not in any of the subregions; if False, only aggregate variables over subregions
- col_apply(col, func, *args, **kwargs)[source]¶
Apply a function to a column of data or meta
- Parameters
- col: str
column in either data or meta dataframes
- func: function
function to apply
- compute_bias(name, method, axis)[source]¶
Compute the bias weights and add to ‘meta’
- Parameters
- namestr
Column name in the ‘meta’ dataframe
- methodstr
Method to compute the bias weights, see the notes
- axisstr
Index dimensions on which to apply the method
Notes
The following methods are implemented:
“count”: use the inverse of the number of scenarios grouped by axis names.
Using the following method on an IamDataFrame with three scenarios
df.compute_bias(name="bias-weight", method="count", axis="scenario")
results in the following column to be added to df.meta:
model
scenario
bias-weight
model_a
scen_a
0.5
model_a
scen_b
1
model_b
scen_a
0.5
- convert_unit(current, to, factor=None, registry=None, context=None, inplace=False)[source]¶
Convert all timeseries data having current units to new units.
If factor is given, existing values are multiplied by it, and the to units are assigned to the ‘unit’ column.
Otherwise, the
pint
package is used to convert from current -> to units without an explicit conversion factor. Pint natively handles conversion between any standard (SI) units that have compatible dimensionality, such as exajoule to terawatt-hours,EJ -> TWh
, or tonne per year to gram per second,t / yr -> g / sec
.The default registry includes additional unit definitions relevant for integrated assessment models and energy systems analysis, via the iam-units package. This registry can also be accessed directly, using:
from iam_units import registry
When using this registry, current and to may contain the symbols of greenhouse gas (GHG) species, such as ‘CO2e’, ‘C’, ‘CH4’, ‘N2O’, ‘HFC236fa’, etc., as well as lower-case aliases like ‘co2’ supported by
pyam
. In this case, context must be the name of a specific global warming potential (GWP) metric supported byiam_units
, e.g. ‘AR5GWP100’ (optionally prefixed by ‘gwp_’, e.g. ‘gwp_AR5GWP100’).Rows with units other than current are not altered.
- Parameters
- currentstr
Current units to be converted.
- tostr
New unit (to be converted to) or symbol for target GHG species. If only the GHG species is provided, the units (e.g.
Mt / year
) will be the same as current, and an expression combining units and species (e.g. ‘Mt CO2e / yr’) will be placed in the ‘unit’ column.- factorvalue, optional
Explicit factor for conversion without pint.
- registry
pint.UnitRegistry
, optional Specific unit registry to use for conversion. Default: the iam-units registry.
- contextstr or
pint.Context
, optional (Name of) the context to use in conversion. Required when converting between GHG species using GWP metrics, unless the species indicated by current and to are the same.
- inplacebool, optional
Whether to return a new IamDataFrame.
- Returns
- Raises
- pint.UndefinedUnitError
if attempting a GWP conversion but context is not given.
- pint.DimensionalityError
without factor, when current and to are not compatible units.
- copy()[source]¶
Make a deepcopy of this object
See
copy.deepcopy()
for details.
- property data¶
Return the timeseries data as a long
pandas.DataFrame
- property dimensions¶
Return the list of data columns (index names & data coordinates)
- divide(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)[source]¶
Divide the timeseries data items a and b along an axis
This function computes a / b. If a or b are lists, the method applies
pandas.groupby().sum()
on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.- Parameters
- a, bstr, list of str or a number
Items to be used for the division.
- namestr
Name of the computed timeseries data on the axis.
- axisstr, optional
Axis along which to compute.
- fillnadict or scalar, optional
Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form
{arg: default}
.- ignore_unitsbool or str, optional
Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).
- appendbool, optional
Whether to append aggregated timeseries data to this instance.
- Returns
IamDataFrame
or NoneComputed timeseries data or None if append=True.
See also
Notes
This function uses the
pint
package and theiam-units
registry (read the docs) to handle units.pyam
will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units'{:~}'.format(u)
(see here for more information).As a result, the notation of returned units may differ from the input format. For example, the unit
EJ/yr
may be reformatted toEJ / a
.
- downscale_region(variable, region='World', subregions=None, proxy=None, weight=None, append=False)[source]¶
Downscale a timeseries to a number of subregions
- Parameters
- variablestr or list of str
variable(s) to be downscaled
- regionstr, optional
region from which data will be downscaled
- subregionslist of str, optional
list of subregions, defaults to all regions other than region (if using proxy) or region index (if using weight)
- proxystr, optional
variable (within the
IamDataFrame
) to be used as proxy for regional downscaling- weightclass:pandas.DataFrame, optional
dataframe with time dimension as columns (year or
datetime.datetime
) and regions[, model, scenario] as index- appendbool, optional
append the downscaled timeseries to self and return None, else return downscaled data as new IamDataFrame
- property empty¶
Indicator whether this object is empty
- equals(other)[source]¶
Test if two objects contain the same data and meta indicators
This function allows two IamDataFrame instances to be compared against each other to see if they have the same timeseries data and meta indicators. nan’s in the same location of the meta table are considered equal.
- Parameters
- otherIamDataFrame
the other
IamDataFrame
to be compared with self
- export_meta(excel_writer, sheet_name='meta')[source]¶
Write the ‘meta’ indicators of this object to an Excel sheet
- Parameters
- excel_writerstr, path object or ExcelWriter object
any valid string path,
pathlib.Path
orpandas.ExcelWriter
- sheet_namestr
name of sheet which will contain dataframe of ‘meta’ indicators
- filter(keep=True, inplace=False, **kwargs)[source]¶
Return a (copy of a) filtered (downselected) IamDataFrame
- Parameters
- keepbool, optional
keep all scenarios satisfying the filters (if True) or the inverse
- inplacebool, optional
if True, do operation inplace and return None
- filters by kwargs:
- The following columns are available for filtering:
‘meta’ columns: filter by string value of that column
‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’: string or list of strings, where * can be used as a wildcard
‘level’: the maximum “depth” of IAM variables (number of ‘|’) (excluding the strings given in the ‘variable’ argument)
- ‘year’: takes an integer (int/np.int64), a list of integers or
a range. Note that the last year of a range is not included,
so range(2010, 2015) is interpreted as [2010, …, 2014]
arguments for filtering by datetime.datetime or np.datetime64 (‘month’, ‘hour’, ‘time’)
‘regexp=True’ disables pseudo-regexp syntax in pattern_match()
- get_data_column(column)[source]¶
Return a column from the timeseries data in long format
Equivalent to
IamDataFrame.data[column]
.- Parameters
- columnstr
The column name.
- Returns
- pd.Series
- head(*args, **kwargs)[source]¶
Identical to
pandas.DataFrame.head()
operating on data
- property index¶
Return all model-scenario combinations as
pandas.MultiIndex
The index allows to loop over the available model-scenario combinations using:
for model, scenario in df.index: ...
- info(n=80, meta_rows=5, memory_usage=False)[source]¶
Print a summary of the object index dimensions and meta indicators
- Parameters
- nint
The maximum line length
- meta_rowsint
The maximum number of meta indicators printed
- interpolate(time, inplace=False, **kwargs)[source]¶
Interpolate missing values in the timeseries data
This method uses
pandas.DataFrame.interpolate()
, which applies linear interpolation by default- Parameters
- timeint or datetime, or list-like thereof
Year or
datetime.datetime
to be interpolated. This must match the datetime/year format of self.- inplacebool, optional
if True, do operation inplace and return None
- kwargs
passed to
pandas.DataFrame.interpolate()
- load_meta(path, sheet_name='meta', ignore_conflict=False, *args, **kwargs)[source]¶
Load ‘meta’ indicators from file
- Parameters
- pathstr,
pathlib.Path
orpandas.ExcelFile
A valid path or instance of an xlsx or csv file
- sheet_namestr, optional
Name of the sheet to be parsed (if xlsx)
- ignore_conflictbool, optional
If True, values in path take precedence over existing meta. If False, raise an error in case of conflicts.
- kwargs
Passed to
pandas.read_excel()
orpandas.read_csv()
- pathstr,
- map_regions(map_col, agg=None, copy_col=None, fname=None, region_col=None, remove_duplicates=False, inplace=False)[source]¶
Plot regional data for a single model, scenario, variable, and year
see pyam.plotting.region_plot() for all available options
- Parameters
- map_colstr
The column used to map new regions to. Common examples include iso and 5_region.
- aggstr, optional
Perform a data aggregation. Options include: sum.
- copy_colstr, optional
Copy the existing region data into a new column for later use.
- fnamestr, optional
Use a non-default region mapping file
- region_colstring, optional
Use a non-default column name for regions to map from.
- remove_duplicatesbool, optional
If there are duplicates in the mapping from one regional level to another, then remove these duplicates by counting the most common mapped value. This option is most useful when mapping from high resolution (e.g., model regions) to low resolution (e.g., 5_region).
- inplacebool, optional
if True, do operation inplace and return None
- property model¶
Return the list of (unique) model names
- multiply(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)[source]¶
Multiply timeseries data items a and b along an axis
This function computes a * b. If a or b are lists, the method applies
pandas.groupby().sum()
on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.- Parameters
- a, bstr, list of str or a number
Items to be used for the division.
- namestr
Name of the computed timeseries data on the axis.
- axisstr, optional
Axis along which to compute.
- fillnadict or scalar, optional
Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form
{arg: default}
.- ignore_unitsbool or str, optional
Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).
- appendbool, optional
Whether to append aggregated timeseries data to this instance.
- Returns
IamDataFrame
or NoneComputed timeseries data or None if append=True.
Notes
This function uses the
pint
package and theiam-units
registry (read the docs) to handle units.pyam
will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units'{:~}'.format(u)
(see here for more information).As a result, the notation of returned units may differ from the input format. For example, the unit
EJ/yr
may be reformatted toEJ / a
.
- normalize(inplace=False, **kwargs)[source]¶
Normalize data to a specific data point
Note: Currently only supports normalizing to a specific time.
- Parameters
- inplacebool, optional
if
True
, do operation inplace and return None- kwargs
the column and value on which to normalize (e.g., year=2005)
- pivot_table(index, columns, values='value', aggfunc='count', fill_value=None, style=None)[source]¶
Returns a pivot table
- Parameters
- indexstr or list of str
rows for Pivot table
- columnsstr or list of str
columns for Pivot table
- valuesstr, default ‘value’
dataframe column to aggregate or count
- aggfuncstr or function, default ‘count’
function used for aggregation, accepts ‘count’, ‘mean’, and ‘sum’
- fill_valuescalar, default None
value to replace missing values with
- stylestr, default None
output style for pivot table formatting accepts ‘highlight_not_max’, ‘heatmap’
- property region¶
Return the list of (unique) regions
- rename(mapping=None, inplace=False, append=False, check_duplicates=True, **kwargs)[source]¶
Rename any index dimension or data coordinate.
When renaming models or scenarios, the uniqueness of the index must be maintained, and the function will raise an error otherwise.
Renaming is only applied to any data row that matches for all columns given in mapping. Renaming can only be applied to the model and scenario columns, or to other data coordinates simultaneously.
- Parameters
- mappingdict or kwargs
mapping of column name to rename-dictionary of that column
dict(<column_name>: {<current_name_1>: <target_name_1>, <current_name_2>: <target_name_2>})
or kwargs as column_name={<current_name_1>: <target_name_1>, …}
- inplacebool, optional
Do operation inplace and return None.
- appendbool, optional
Whether to append aggregated timeseries data to this instance (if inplace=True) or to a returned new instance (if inplace=False).
- check_duplicatesbool, optional
Check whether conflicts exist after renaming of timeseries data coordinates. If True, raise a ValueError; if False, rename and merge with
groupby().sum()
.
- Returns
IamDataFrame
or NoneAggregated timeseries data as new object or None if inplace=True.
- require_variable(variable, unit=None, year=None, exclude_on_fail=False)[source]¶
Check whether all scenarios have a required variable
- Parameters
- variablestr
required variable
- unitstr, default None
name of unit (optional)
- yearint or list, default None
check whether the variable exists for ANY of the years (if a list)
- exclude_on_failbool, default False
flag scenarios missing the required variables as exclude: True
- property scenario¶
Return the list of (unique) scenario names
- set_meta(meta, name=None, index=None)[source]¶
Add meta indicators as pandas.Series, list or value (int/float/str)
- Parameters
- metapandas.Series, list, int, float or str
column to be added to ‘meta’ (by [‘model’, ‘scenario’] index if possible)
- namestr, optional
meta column name (defaults to meta pandas.Series.name); either meta.name or the name kwarg must be defined
- indexIamDataFrame, pandas.DataFrame or pandas.MultiIndex, optional
index to be used for setting meta column ([‘model’, ‘scenario’])
- set_meta_from_data(name, method=None, column='value', **kwargs)[source]¶
Add meta indicators from downselected timeseries data of self
- Parameters
- namestr
column name of the ‘meta’ table
- methodfunction, optional
method for aggregation (e.g.,
numpy.max
); required if downselected data do not yield unique values- columnstr, optional
the column from data to be used to derive the indicator
- kwargs
passed to
filter()
for downselected data
- subtract(a, b, name, axis='variable', fillna=None, ignore_units=False, append=False)[source]¶
Compute the difference of timeseries data items a and b along an axis
This function computes a - b. If a or b are lists, the method applies
pandas.groupby().sum()
on each group. If either a or b are not defined for a row and fillna is not specified, no value is computed for that row.- Parameters
- a, bstr, list of str or a number
Items to be used for the subtraction.
- namestr
Name of the computed timeseries data on the axis.
- axisstr, optional
Axis along which to compute.
- fillnadict or scalar, optional
Value to fill holes when rows are not defined for either a or b. Can be a scalar or a dictionary of the form
{arg: default}
.- ignore_unitsbool or str, optional
Perform operation on values without considering units. Set units of returned data to unknown (if True) or the value of ignore_units (if str).
- appendbool, optional
Whether to append aggregated timeseries data to this instance.
- Returns
IamDataFrame
or NoneComputed timeseries data or None if append=True.
Notes
This function uses the
pint
package and theiam-units
registry (read the docs) to handle units.pyam
will keep notation consistent with the input format (if possible) and otherwise uses abbreviated units'{:~}'.format(u)
(see here for more information).As a result, the notation of returned units may differ from the input format. For example, the unit
EJ/yr
may be reformatted toEJ / a
.
- swap_time_for_year(subannual=False, inplace=False)[source]¶
Convert the time dimension to year (as integer).
- Parameters
- subannualbool, str or func, optional
Merge non-year components of the “time” domain as new column “subannual”. Apply
strftime()
on the values of the “time” domain using subannual (if a string) or “%m-%d %H:%M%z” (if True). If it is a function, apply the function on the values of the “time” domain.- inplacebool, optional
If True, do operation inplace and return None.
- Returns
IamDataFrame
or NoneObject with altered time domain or None if inplace=True.
- Raises
- ValueError
“time” is not a column of self.data
See also
- swap_year_for_time(inplace=False)[source]¶
Convert the year and subannual dimensions to time (as datetime).
The method applies
dateutil.parser.parse()
on the combined columns year and subannual:dateutil.parser.parse([f"{y}-{s}" for y, s in zip(year, subannual)])
- Parameters
- inplacebool, optional
If True, do operation inplace and return None.
- Returns
IamDataFrame
or NoneObject with altered time domain or None if inplace=True.
- Raises
- ValueError
“year” or “subannual” are not a column of self.data
See also
- tail(*args, **kwargs)[source]¶
Identical to
pandas.DataFrame.tail()
operating on data
- timeseries(iamc_index=False)[source]¶
Returns data as
pandas.DataFrame
in wide format- Parameters
- iamc_indexbool, optional
if True, use [‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’]; else, use all ‘data’ columns
- Raises
- ValueError
IamDataFrame is empty
- ValueError
reducing to IAMC-index yields an index with duplicates
- to_csv(path, iamc_index=False, **kwargs)[source]¶
Write timeseries data of this object to a csv file
- Parameters
- pathstr or path object
file path or
pathlib.Path
- iamc_indexbool, default False
if True, use [‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’]; else, use all ‘data’ columns
- to_datapackage(path)[source]¶
Write object to a frictionless Data Package
More information: https://frictionlessdata.io
Returns the saved
datapackage.Package
(read the docs). When adding metadata (descriptors), please follow the template defined by https://github.com/OpenEnergyPlatform/metadata- Parameters
- pathstring or path object
any valid string path or
pathlib.Path
- to_excel(excel_writer, sheet_name='data', iamc_index=False, include_meta=True, **kwargs)[source]¶
Write object to an Excel spreadsheet
- Parameters
- excel_writerstr, path object or ExcelWriter object
any valid string path,
pathlib.Path
orpandas.ExcelWriter
- sheet_namestring
name of sheet which will contain
timeseries()
data- iamc_indexbool, default False
if True, use [‘model’, ‘scenario’, ‘region’, ‘variable’, ‘unit’]; else, use all ‘data’ columns
- include_metaboolean or string
if True, write ‘meta’ to an Excel sheet name ‘meta’ (default); if this is a string, use it as sheet name
- property unit¶
Return the list of (unique) units
- property unit_mapping¶
Return a dictionary of variables to (list of) correspoding units
- validate(criteria={}, exclude_on_fail=False)[source]¶
Validate scenarios using criteria on timeseries values
Returns all scenarios which do not match the criteria and prints a log message, or returns None if all scenarios match the criteria.
When called with exclude_on_fail=True, scenarios not satisfying the criteria will be marked as exclude=True.
- Parameters
- criteriadict
- dictionary with variable keys and validation mappings
(‘up’ and ‘lo’ for respective bounds, ‘year’ for years)
- exclude_on_failbool, optional
flag scenarios failing validation as exclude: True
- Returns
pandas.DataFrame
All data points that do not satisfy the criteria.
- None
If all scenarios satisfy the criteria.
- property variable¶
Return the list of (unique) variables