Introduction to Pandas Data Structure

Series

A Series is a one-dimensional array-like object containing a sequence of values (of similar types to NumPy types) and an associated array of data labels, called its index. The simplest Series is formed from only an array of data:

Input[1]: obj = pd.Series([4, 7, -5, 3])
Input[2]: obj
Output: 
0 4
1 7
2 -5
3 3
dtype: int64

The string representation of a Series displayed interactively shows the index on the left and the values on the right. Since we did not specify an index for the data, a default one consisting of the integers 0 through N – 1 (where N is the length of the data) is created. You can get the array representation and index object of the Series via its values and index attributes, respectively:

Input[3]: obj.values
Output: array([ 4, 7, -5, 3])
Input[4]: obj.index
Output: RangeIndex(start=0, stop=4, step=1)

Often it will be desirable to create a Series with an index identifying each data point with a label:

Input[5]: obj2 = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])
Input[6]: obj2
Output:
d 4
b 7
a -5
c 3
dtype: int64
Input[7]: obj2.index
Output: Index(['d', 'b', 'a', 'c'], dtype='object')

Compared with NumPy arrays, you can use labels in the index when selecting single values or a set of values:

Input[8]: obj2['a']
Output: -5
Input[9]: obj2['d'] = 6
Input[10]: obj2[['c', 'a', 'd']]
Output:
c 3
a -5
d 6
dtype: int64

Here [‘c’, ‘a’, ‘d’] is interpreted as a list of indices, even though it contains strings instead of integers.

Using NumPy functions or NumPy-like operations, such as filtering with a boolean array, scalar multiplication, or applying math functions, will preserve the index-value link:

Input[11]: obj2[obj2 > 0]
Output:
d 6
b 7
c 3
dtype: int64

Input[12]: obj2 * 2
Output:
d 12
b 14
a -10
c 6
dtype: int64

Input[13]: np.exp(obj2)
Output:
d 403.428793
b 1096.633158
a 0.006738
c 20.085537
dtype: float64

Another way to think about a Series is as a fixed-length, ordered dict, as it is a mapping of index values to data values. It can be used in many contexts where you might use a dict:

Input[14]: 'b' in obj2
Output: True
Input[15]: 'e' in obj2
Output: False

Should you have data contained in a Python dict, you can create a Series from it by passing the dict:

Input[16]: sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}
Input[17]: obj3 = pd.Series(sdata)
Input[18]: obj3
Output:
Ohio 35000
Oregon 16000
Texas 71000
Utah 5000
dtype: int64

When you are only passing a dict, the index in the resulting Series will have the dict’s keys in sorted order. You can override this by passing the dict keys in the order you want them to appear in the resulting Series:

Input[19]: states = ['California', 'Ohio', 'Oregon', 'Texas']
Input[20]: obj4 = pd.Series(sdata, index=states)
Input[21]: obj4
Output:
California NaN
Ohio 35000.0
Oregon 16000.0
Texas 71000.0
dtype: float64

Here, three values found in sdata were placed in the appropriate locations, but since no value for ‘California’ was found, it appears as NaN (not a number), which is considered in pandas to mark missing or NA values. Since ‘Utah’ was not included in states, it is excluded from the resulting object.
I will use the terms “missing” or “NA” interchangeably to refer to missing data. The isnull and notnull functions in pandas should be used to detect missing data:

Input[22]: pd.isnull(obj4)
Output:
California True
Ohio False
Oregon False
Texas False
dtype: bool

Input[23]: pd.notnull(obj4)
Output:
California False
Ohio True
Oregon True
Texas True
dtype: bool

Series also has these as instance methods:

Input[14]: obj4.isnull()
Output:
California True
Ohio False
Oregon False
Texas False
dtype: bool

A useful Series feature for many applications is that it automatically aligns by index label in arithmetic operations:

Input[25]: obj3
Output:
Ohio 35000
Oregon 16000
Texas 71000
Utah 5000
dtype: int64

Input[26]: obj4
Output:
California NaN
Ohio 35000.0
Oregon 16000.0
Texas 71000.0
dtype: float64

Input[27]: obj3 + obj4
Output:
California NaN
Ohio 70000.0
Oregon 32000.0
Texas 142000.0
Utah NaN
dtype: float64

Both the Series object itself and its index have a name attribute, which integrates with other key areas of pandas functionality:

Input[28]: obj4.name = 'population'
Input[29]: obj4.index.name = 'state'
Input[30]: obj4
Output:
state
California NaN
Ohio 35000.0
Oregon 16000.0
Texas 71000.0
Name: population, dtype: float64

A Series’s index can be altered in-place by assignment:

Input[31]: obj
Output:
0 4
1 7
2 -5
3 3
dtype: int64

Input[32]: obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']
Input[33]: obj
Output:
Bob 4
Steve 7
Jeff -5
Ryan 3
dtype: int64

DataFrame

A DataFrame represents a rectangular table of data and contains an ordered collection of columns, each of which can be a different value type (numeric, string, boolean, etc.). The DataFrame has both a row and column index; it can be thought of as a dict of Series all sharing the same index. Under the hood, the data is stored as one or more two-dimensional blocks rather than a list, dict, or some other collection of one-dimensional arrays.

There are many ways to construct a DataFrame, though one of the most common is from a dict of equal-length lists or NumPy arrays:

data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
        'year': [2000, 2001, 2002, 2001, 2002, 2003],
        'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)

The resulting DataFrame will have its index assigned automatically as with Series, and the columns are placed in sorted order:

Input[35]: frame
Output:
	state	year	pop
0	Ohio	2000	1.5
1	Ohio	2001	1.7
2	Ohio	2002	3.6
3	Nevada	2001	2.4
4	Nevada	2002	2.9
5	Nevada	2003	3.2

For large DataFrames, the head method selects only the first five rows:

Input[36]: frame.head()
Output:
	state	year	pop
0	Ohio	2000	1.5
1	Ohio	2001	1.7
2	Ohio	2002	3.6
3	Nevada	2001	2.4
4	Nevada	2002	2.9

If you specify a sequence of columns, the DataFrame’s columns will be arranged in that order:

Input[37]: pd.DataFrame(data, columns=['year', 'state', 'pop'])
Output:
	year	state	pop
0	2000	Ohio	1.5
1	2001	Ohio	1.7
2	2002	Ohio	3.6
3	2001	Nevada	2.4
4	2002	Nevada	2.9
5	2003	Nevada	3.2

If you pass a column that isn’t contained in the dict, it will appear with missing values in the result:

Input[38]: frame2 = pd.DataFrame(data, columns=['year', 'state', 'pop', 'debt'],
                                     index=['one', 'two', 'three', 'four','five', 'six'])

Input[39]: frame2
Output:
	year	state	pop	debt
one	2000	Ohio	1.5	NaN
two	2001	Ohio	1.7	NaN
three	2002	Ohio	3.6	NaN
four	2001	Nevada	2.4	NaN
five	2002	Nevada	2.9	NaN
six	2003	Nevada	3.2	NaN

Input[40]: frame2.columns
Output: Index(['year', 'state', 'pop', 'debt'], dtype='object')

A column in a DataFrame can be retrieved as a Series either by dict-like notation or by attribute:

Input[41]: frame2['state']
Output:
one Ohio
two Ohio
three Ohio
four Nevada
five Nevada
six Nevada
Name: state, dtype: object

Input[42]: frame2.year
Output:
one 2000 
two 2001
three 2002
four 2001
five 2002
six 2003
Name: year, dtype: int64

NOTE:-Attribute-like access (e.g., frame2.year) and tab completion of column names in IPython is provided as a convenience. frame2[column] works for any column name, but frame2.column only works when the column name is a valid Python variable name.

Note that the returned Series have the same index as the DataFrame, and their name attribute has been appropriately set.

Rows can also be retrieved by position or name with the special loc attribute.

Input[43]: frame2.loc['three']
Output:
year 2002
state Ohio
pop 3.6
debt NaN
Name: three, dtype: object

Columns can be modified by assignment. For example, the empty ‘debt’ column could be assigned a scalar value or an array of values:

Input[44]: frame2['debt'] = 16.5
Input[45]: frame2
Output:
	year	state	pop	debt
one	2000	Ohio	1.5	16.5
two	2001	Ohio	1.7	16.5
three	2002	Ohio	3.6	16.5
four	2001	Nevada	2.4	16.5
five	2002	Nevada	2.9	16.5
six	2003	Nevada	3.2	16.5

Input[46]: frame2['debt'] = np.arange(6.)
Input[47]: frame2
Output:
	year	state	pop	debt
one	2000	Ohio	1.5	0.0
two	2001	Ohio	1.7	1.0
three	2002	Ohio	3.6	2.0
four	2001	Nevada	2.4	3.0
five	2002	Nevada	2.9	4.0
six	2003	Nevada	3.2	5.0

When you are assigning lists or arrays to a column, the value’s length must match the length of the DataFrame. If you assign a Series, its labels will be realigned exactly to the DataFrame’s index, inserting missing values in any holes:

Input[48]: val = pd.Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])
Input[49]: frame2['debt'] = val
Input[50]: frame2
Output:
	year	state	pop	debt
one	2000	Ohio	1.5	NaN
two	2001	Ohio	1.7	-1.2
three	2002	Ohio	3.6	NaN
four	2001	Nevada	2.4	-1.5
five	2002	Nevada	2.9	-1.7
six	2003	Nevada	3.2	NaN

Assigning a column that doesn’t exist will create a new column. The del keyword will delete columns as with a dict.
As an example of del, I first add a new column of boolean values where the state column equals ‘Ohio’:

Input[51]: frame2['eastern'] = frame2.state == 'Ohio'
Input[52]: frame2
Output:
	year	state	pop	debt	eastern
one	2000	Ohio	1.5	NaN	True
two	2001	Ohio	1.7	-1.2	True
three	2002	Ohio	3.6	NaN	True
four	2001	Nevada	2.4	-1.5	False
five	2002	Nevada	2.9	-1.7	False
six	2003	Nevada	3.2	NaN	False

NOTE:-New columns cannot be created with the frame2.eastern syntax.

The del method can then be used to remove this column:

Input[53]: del frame2['eastern']
Input[54]: frame2.columns
Output: Index(['year', 'state', 'pop', 'debt'], dtype='object')

NOTE:-The column returned from indexing a DataFrame is a view on the underlying data, not a copy. Thus, any in-place modifications to the Series will be reflected in the DataFrame. The column can be explicitly copied with the Series’s copy method.

Another common form of data is a nested dict of dicts:

Input[55]: pop = {'Nevada': {2001: 2.4, 2002: 2.9},
        'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}

If the nested dict is passed to the DataFrame, pandas will interpret the outer dict keys as the columns and the inner keys as the row indices:

Input[56]: frame3 = pd.DataFrame(pop)
Input[57]: frame3
Output:
	Nevada	Ohio
2001	2.4	1.7
2002	2.9	3.6
2000	NaN	1.5

You can transpose the DataFrame (swap rows and columns) with similar syntax to a NumPy array

Input[58]: frame3.T
Output:
	2001	2002	2000
Nevada	2.4	2.9	NaN
Ohio	1.7	3.6	1.5

The keys in the inner dicts are combined and sorted to form the index in the result. This isn’t true if an explicit index is specified:

Input[59]: pd.DataFrame(pop, index=[2001, 2002, 2003])
Output:
	Nevada	Ohio
2001	2.4	1.7
2002	2.9	3.6
2003	NaN	NaN

Dicts of Series are treated in much the same way:

Input[60]: pdata = {'Ohio': frame3['Ohio'][:-1],
                  'Nevada': frame3['Nevada'][:2]}
Input[61]: pd.DataFrame(pdata)
Output:
	Ohio	Nevada
2001	1.7	2.4
2002	3.6	2.9

For a complete list of things you can pass the DataFrame constructor

Types	Notes
2D ndarray	A matrix of data, passing optional row and column labels
dict of arrays, lists, or tuples	Each sequence becomes a column in the DataFrame; all sequences must be the same length
NumPy structured/record array	Treated as the “dict of arrays” case
dict of Series	Each value becomes a column; indexes from each Series are unioned together to form the result’s row index if no explicit index is passed
dict of dicts	Each inner dict becomes a column; keys are unioned to form the row index as in the “dict of Series” case
List of dicts or Series	Each item becomes a row in the DataFrame; union of dict keys or Series indexes become the
List of lists or tuples	Treated as the “2D ndarray” case
Another DataFrame	The DataFrame’s indexes are used unless different ones are passed
NumPy MaskedArray	Like the “2D ndarray” case except masked values become NA/missing in the DataFrame result

If a DataFrame’s index and columns have their name attributes set, these will also be displayed:

Input[62]: frame3.index.name = 'year'; frame3.columns.name = 'state'
Input[63]: frame3
Output:
state	Nevada	Ohio
year		
2001	2.4	1.7
2002	2.9	3.6
2000	NaN	1.5

As with Series, the values attribute returns the data contained in the DataFrame as a two-dimensional ndarray:

Input[64]: frame3.values
Output:
array([[ nan, 1.5],
       [ 2.4, 1.7],
       [ 2.9, 3.6]])

If the DataFrame’s columns are different dtypes, the dtype of the values array will be chosen to accommodate all of the columns:

Input[65]: frame2.values
Output:
array([[2000, 'Ohio', 1.5, nan],
       [2001, 'Ohio', 1.7, -1.2],
       [2002, 'Ohio', 3.6, nan],
       [2001, 'Nevada', 2.4, -1.5],
       [2002, 'Nevada', 2.9, -1.7],
       [2003, 'Nevada', 3.2, nan]], dtype=object)

Index Objects

pandas’s Index objects are responsible for holding the axis labels and other metadata (like the axis name or names). Any array or other sequence of labels you use when constructing a Series or DataFrame is internally converted to an Index:

Input[66]: obj = pd.Series(range(3), index=['a', 'b', 'c'])
Input[67]: index = obj.index
Input[68]: index
Output: Index(['a', 'b', 'c'], dtype='object')
In [69]: index[1:]
Output: Index(['b', 'c'], dtype='object')

Index objects are immutable and thus can’t be modified by the user:

index[1] = 'd'


TypeError: Index does not support mutable operations

Immutability makes it safer to share Index objects among data structures:

Input[70]: labels = pd.Index(np.arange(3))
Input[71]: labels
Output: Int64Index([0, 1, 2], dtype='int64')
Input[72]: obj2 = pd.Series([1.5, -2.5, 0], index=labels)
Input[73]: obj2
Output:
0     1.5
1    -2.5
2     0.0
dtype: float64
Input[74]: obj2.index is labels
Output: True

In addition to being array-like, an Index also behaves like a fixed-size set:

Input[75]: frame3
Output:
state  Nevada  Ohio
year
2000    NaN    1.5
2001    2.4    1.7
2002    2.9    3.6

Input[76]: frame3.columns
Output: Index(['Nevada', 'Ohio'], dtype='object', name='state')

Input[77]: 'Ohio' in frame3.columns
Output: True

In [78]: 2003 in frame3.index
Output: False
Unlike Python sets, a pandas Index can contain duplicate labels:

Input[79]: dup_labels = pd.Index(['foo', 'foo', 'bar', 'bar'])
Input[80]: dup_labels
Output: Index(['foo', 'foo', 'bar', 'bar'], dtype='object')

Selections with duplicate labels will select all occurrences of that label.
Each Index has a number of methods and properties for set logic, which answer other common questions about the data it contains.

Method	Description
append	Concatenate with additional Index objects, producing a new Index
difference	Compute set difference as an Index
intersection	Compute set intersection
union	Compute set union
isin	Compute boolean array indicating whether each value is contained in the passed collection
delete	Compute new Index with element at index i deleted
drop	Compute new Index by deleting passed values
insert	Compute new Index by inserting element at index i
is_monotonic	Returns True if each element is greater than or equal to the previous element
is_unique	Returns True if the Index has no duplicate values
unique	Compute the array of unique values in the Index