Pandas实现哑变量

Pandas哑变量生成

哑变量又叫做虚拟变量，虚设变量或者名义变量，是人为设定的用于将分类变量引入回归模型中的方法。比如学历、职业、性别等分类变量的数据是不能量化的，通过构造0和1的哑变量可以考察定性因素（分类变量）对因变量的影响。

哑变量一般在回归的相关模型中经常使用。在虚拟变量的设置中：表示的基础类型、肯定类型取值为1；如果是比较类型，否定类型则取值为0。

在实际的数据处理中，通过独热码one-hot来实现哑变量。Pandas中的get_dummies函数能够实现此功能。

get_dummies使用

pandas.get_dummies(data,  # 待处理数据
                   prefix=None, #
                   prefix_sep='_', # 连接符
                   dummy_na=False,  # 是否显示控制
                   columns=None, # 指定字段
                   sparse=False, # 是否表示为稀疏矩阵
                   drop_first=False, # 是否删除生成后的第一个字段
                   dtype=None)

import pandas as pd
import numpy as np

from sklearn.preprocessing import OneHotEncoder

s = pd.Series(list("abadc"))
s

0    a
1    b
2    a
3    d
4    c
dtype: object

pd.get_dummies(s)

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

前缀处理-prefix

pd.get_dummies(s)

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(s, prefix="col")  # 统一加上前缀col，默认连接符是_

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

连接符

pd.get_dummies(s, prefix="col", prefix_sep=".")  # 统一加上前缀col，连接符是.

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

空值处理

s1 = pd.Series(["a","b",np.nan,"c"])
s1

0      a
1      b
2    NaN
3      c
dtype: object

pd.get_dummies(s1, dummy_na=False)  # 默认是False

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(s1, dummy_na=True) # 显示空值

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

删除第一个字段

pd.get_dummies(s1)

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(s1, drop_first=True)

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

指定字段类型dtype

pd.get_dummies(s1, dtype="float")

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

案例1-两种分类

df = pd.DataFrame({
    "id":["ID1","ID2","ID3","ID4","ID5","ID6"],
    "sex":["Female","Male","Female","Male","Male","Female"],
    "amount":[2000,2500,1800,1900,2000,2600]
})

df

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(df["sex"])

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

结果：从sex变量延伸出两个变量Female和Male，这两个变量就是sex中的不同取值。

当原数据中出现了Female，则哑变量Female取值为1，否则为0；Male是一样的

pd.get_dummies(df["sex"], prefix="sex")

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

# 指定对sex执行独热码

pd.get_dummies(df, columns=["sex"])

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

案例2-多种分类

df1 = pd.DataFrame({
    "id":["ID1","ID2","ID3","ID4","ID5","ID6"],
    "sex":["Female","Male","Female","Male","Male","Female"],
    "education":["高中","本科","本科","研究生","本科","研究生"],
    "amount":[1000,2500,1800,4900,2000,3600]
})

df1

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(df1["education"])

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(df1, columns=["education"])

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

案例3-多个字段

df1

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

pd.get_dummies(df1, columns=["sex","education"])

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

OneHotEncoder()使用

官网案例

enc = OneHotEncoder()
enc.fit([[0,0,3],
         [1,1,0],
         [0,2,1],
         [1,0,2]])  #这里一共有4个数据，3种特征

array = enc.transform([[0,1,3]]).toarray()

print(array)

[[1. 0. 0. 1. 0. 0. 0. 0. 1.]]

array = enc.transform([[1,2,1]]).toarray()

print(array)

[[0. 1. 0. 0. 1. 0. 1. 0. 0.]]

实际案例

df1

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

enc = OneHotEncoder()

enc.fit(df1[["sex", "education"]])

OneHotEncoder()

enc.categories_

[array(['Female', 'Male'], dtype=object),
 array(['本科', '研究生', '高中'], dtype=object)]

# 测试案例

enc.transform([["Male","研究生"]]).toarray()

/Applications/downloads/anaconda/anaconda3/lib/python3.7/site-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but OneHotEncoder was fitted with feature names
  "X does not have valid feature names, but"





array([[0., 1., 0., 1., 0.]])