Python进程管理

星哥玩云

发布于 2022-09-08 13:53:31

5220

发布于 2022-09-08 13:53:31

文章被收录于专栏：开源部署

一、多任务原理

概念现代操作系统比如Mac OS X，UNIX，Linux，Windows等，都是支持“多任务”的操作系统
什么叫多任务？就是操作系统可以同时运行多个任务
单核CPU实现多任务原理

多核CPU实现多任务原理

实现多任务的方式
- 多进程模式启动多个进程，每个进程虽然只有一个线程，但是多个进程可以一起执行多个任务
- 多线程模式启动一个进程，在一个进程的内部启动多个线程，这样多个线程也可以一起执行多个任务
- 多进程+多线程启动多个进程，每个进程再启动多个线程
- 协程
- 多进程+协程

二、进程

1、概念

什么是进程？是计算机中的程序关于某数据集合上的一次运行活动，是系统进行资源分配和调度的基本单位，是操作系统结构的基础。在早期面向进程设计的计算机结构中，进程是程序的基本执行实体；在当代面向线程设计的计算机结构中，进程是线程的容器。程序是指令、数据及其组织形式的描述，进程是程序的实体
对于操作系统一个任务就是一个进程。比方说打开浏览器就是启动一个浏览器的进程，在打开一个记事本就启动一个记事本进程，如果打开两个记事本就启动两个记事本进程
注意进程是系统中程序执行和资源分配的基本单位。每个进程有自己的数据段、代码段和堆栈段

2、使用进程

单进程现象

<span class="hljs-keyword">import</span> time

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run1</span><span class="hljs-params">()</span>:</span>
    <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
        print(<span class="hljs-string">"zutuanxue_com is a good man"</span>)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run2</span><span class="hljs-params">()</span>:</span>
    <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
        print(<span class="hljs-string">"zutuanxue_com is a nice man"</span>)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    run1()
    <span class="hljs-comment"># 不会执行run2()函数，只有上面的run1()结束才能执行run2()</span>
    run2()

启动进程实现多任务

multiprocessing模块

跨平台的多进程模块，提供了一个Process类用来示例化一个进程对象

Process类

作用：创建进程(子进程)

参数	说明
target	指定进程执行的任务
args	给进程函数传递的参数，是一个元组

注意：此时进程被创建，但是不会启动进程执行启动进程：Process对象.start()

示例代码

<span class="hljs-keyword">import</span> time

<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run1</span><span class="hljs-params">(name)</span>:</span>
    <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
        print(<span class="hljs-string">"%s is a good man"</span>%name)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run2</span><span class="hljs-params">()</span>:</span>
    <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
        print(<span class="hljs-string">"zutuanxue_com is a nice man"</span>)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    <span class="hljs-comment"># 程序启动时的进程称为主进程(父进程)</span>
    
    <span class="hljs-comment"># 创建进程并启动</span>
    p = Process(target=run1, args=(<span class="hljs-string">"kaige"</span>,))
    p.start()

    <span class="hljs-comment"># 主进程执行run2()函数</span>
    run2()

主进程负责调度

主进程主要做的是调度相关的工作，一般不负责具体业务逻辑

<span class="hljs-keyword">import</span> time
<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run1</span><span class="hljs-params">()</span>:</span>
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">7</span>):
        print(<span class="hljs-string">"zutuanxue_com is a good man"</span>)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run2</span><span class="hljs-params">(name, word)</span>:</span>
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">5</span>):
        print(<span class="hljs-string">"%s is a %s man"</span>%(name, word))
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    t1 = time.time()

    <span class="hljs-comment"># 创建两个进程分别执行run1、run2</span>
    p1 = Process(target=run1)
    p2 = Process(target=run2, args=(<span class="hljs-string">"kaige"</span>, <span class="hljs-string">"cool"</span>))

    <span class="hljs-comment"># 启动两个进程</span>
    p1.start()
    p2.start()

    <span class="hljs-comment"># 查看耗时</span>
    t2 = time.time()
    print(<span class="hljs-string">"耗时：%.2f"</span>%(t2-t1))

父子进程的先后顺序

主进程的结束不能影响子进程，所以可以等待子进程的结束再结束主进程，等待子进程结束，才能继续运行主进程

<span class="hljs-keyword">import</span> time
<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run1</span><span class="hljs-params">()</span>:</span>
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">7</span>):
        print(<span class="hljs-string">"zutuanxue_com is a good man"</span>)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run2</span><span class="hljs-params">(name, word)</span>:</span>
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">5</span>):
        print(<span class="hljs-string">"%s is a %s man"</span>%(name, word))
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    t1 = time.time()

    p1 = Process(target=run1)
    p2 = Process(target=run2, args=(<span class="hljs-string">"kaige"</span>, <span class="hljs-string">"cool"</span>))

    p1.start()
    p2.start()

    <span class="hljs-comment"># 主进程的结束不能影响子进程，所以可以等待子进程的结束再结束主进程</span>
    <span class="hljs-comment"># 等待子进程结束，才能继续运行主进程</span>
    p1.join()
    p2.join()

    t2 = time.time()
    print(<span class="hljs-string">"耗时：%.2f"</span>%(t2-t1))

3、全局变量在多个子进程中不能共享

<span class="hljs-keyword">import</span> time
<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process

money = <span class="hljs-number">100</span>

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run1</span><span class="hljs-params">()</span>:</span>
    <span class="hljs-comment"># global money # 在进程中无法使用全局变量</span>
    money = <span class="hljs-number">200</span>
    print(<span class="hljs-string">"run1----------------"</span>, money, id(money))
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">5</span>):
        print(<span class="hljs-string">"zutuanxue_com is a good man"</span>)
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run2</span><span class="hljs-params">(name, word)</span>:</span>
    print(<span class="hljs-string">"run2----------------"</span>, money, id(money))
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">5</span>):
        print(<span class="hljs-string">"%s is a %s man"</span>%(name, word))
        time.sleep(<span class="hljs-number">1</span>)

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    t1 = time.time()

    <span class="hljs-comment"># 在创建进程时会将主进程的资源拷贝到子进程中，子进程单独有一份主进程中的数据，相互不影响</span>
    p1 = Process(target=run1)
    p2 = Process(target=run2, args=(<span class="hljs-string">"kaige"</span>, <span class="hljs-string">"cool"</span>))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    print(<span class="hljs-string">"main-------------"</span>, money)

    t2 = time.time()
    print(<span class="hljs-string">"耗时：%.2f"</span>%(t2-t1))

4、启动大量子进程

<span class="hljs-comment"># Pool类：进程池类</span>
<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Pool
<span class="hljs-keyword">import</span> time
<span class="hljs-keyword">import</span> random

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run</span><span class="hljs-params">(index)</span>:</span>
    print(<span class="hljs-string">"子进程 %d 启动"</span>%(index))
    t1 = time.time()
    time.sleep(random.random()* <span class="hljs-number">5</span>+<span class="hljs-number">2</span>)
    t2 = time.time()
    print(<span class="hljs-string">"子进程 %d 结束，耗时：%.2f"</span> % (index, t2-t1))

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    print(<span class="hljs-string">"启动主进程……"</span>)

    <span class="hljs-comment"># 创建10个子进程执行run功能</span>


    <span class="hljs-comment"># 创建进程池对象</span>
    <span class="hljs-comment"># 由于pool的默认值为CPU的核心数，假设有4核心，至少需要5个子进程才能看到效果</span>
    <span class="hljs-comment"># Pool()中的值表示可以同时执行进程的数量</span>
    pool = Pool()
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">1</span>, <span class="hljs-number">7</span>):
        <span class="hljs-comment"># 创建子进程，并将子进程放到进程池中统一管理</span>
        pool.apply_async(run, args=(i,))

    <span class="hljs-comment"># 等待子进程结束</span>
    <span class="hljs-comment"># 关闭进程池：在关闭后就不能再向进程池中添加进程了</span>
    <span class="hljs-comment"># 进程池对象在调用join之前必须先关闭进程池</span>
    pool.close()
    <span class="hljs-comment">#pool对象调用join，主进程会等待进程池中的所有子进程结束才会继续执行主进程</span>
    pool.join()

    print(<span class="hljs-string">"结束主进程……"</span>)

5、单进程与多进程复制文件对比

单进程复制文件

<span class="hljs-keyword">import</span> time

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">copy_file</span><span class="hljs-params">(path, toPath)</span>:</span>
    <span class="hljs-keyword">with</span> open(path, <span class="hljs-string">"rb"</span>) <span class="hljs-keyword">as</span> fp1:
        <span class="hljs-keyword">with</span> open(toPath, <span class="hljs-string">"wb"</span>) <span class="hljs-keyword">as</span> fp2:
            <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
                info = fp1.read(<span class="hljs-number">1024</span>)
                <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> info:
                    <span class="hljs-keyword">break</span>
                <span class="hljs-keyword">else</span>:
                    fp2.write(info)
                    fp2.flush()

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    t1 = time.time()

    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">1</span>, <span class="hljs-number">5</span>):
        path = <span class="hljs-string">r"/Users/zutuanxue_com/Desktop/file/%d.mp4"</span>%i
        toPath = <span class="hljs-string">r"/Users/zutuanxue_com/Desktop/file2/%d.mp4"</span>%i
        copy_file(path, toPath)

    t2 = time.time()
    print(<span class="hljs-string">"单进程耗时：%.2f"</span>%(t2-t1))

多进程复制文件

<span class="hljs-keyword">import</span> time
<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Pool

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">copy_file</span><span class="hljs-params">(path, toPath)</span>:</span>
    <span class="hljs-keyword">with</span> open(path, <span class="hljs-string">"rb"</span>) <span class="hljs-keyword">as</span> fp1:
        <span class="hljs-keyword">with</span> open(toPath, <span class="hljs-string">"wb"</span>) <span class="hljs-keyword">as</span> fp2:
            <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
                info = fp1.read(<span class="hljs-number">1024</span>)
                <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> info:
                    <span class="hljs-keyword">break</span>
                <span class="hljs-keyword">else</span>:
                    fp2.write(info)
                    fp2.flush()

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    t1 = time.time()

    pool = Pool(<span class="hljs-number">4</span>)
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">1</span>, <span class="hljs-number">5</span>):
        path = <span class="hljs-string">r"/Users/zutuanxue_com/Desktop/file/%d.mp4"</span> % i
        toPath = <span class="hljs-string">r"/Users/zutuanxue_com/Desktop/file3/%d.mp4"</span> % i
        pool.apply_async(copy_file, args=(path, toPath))
    pool.close()
    pool.join()

    t2 = time.time()
    print(<span class="hljs-string">"单进程耗时：%.2f"</span>%(t2-t1

6、进程间通信

方式

有名管道
无名管道
队列
共享内存
信号
信号量

代码

<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process
<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Queue
<span class="hljs-keyword">import</span> time

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">product</span><span class="hljs-params">(q)</span>:</span>
    print(<span class="hljs-string">"启动生产子进程……"</span>)
    <span class="hljs-keyword">for</span> data <span class="hljs-keyword">in</span> [<span class="hljs-string">"good"</span>, <span class="hljs-string">"nice"</span>, <span class="hljs-string">"cool"</span>, <span class="hljs-string">"handsome"</span>]:
        time.sleep(<span class="hljs-number">2</span>)
        print(<span class="hljs-string">"生产出：%s"</span>%data)
        <span class="hljs-comment"># 将生产的数据写入队列</span>
        q.put(data)
    print(<span class="hljs-string">"结束生产子进程……"</span>)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">customer</span><span class="hljs-params">(q)</span>:</span>
    print(<span class="hljs-string">"启动消费子进程……"</span>)
    <span class="hljs-keyword">while</span> <span class="hljs-number">1</span>:
        print(<span class="hljs-string">"等待生产者生产数据"</span>)
        <span class="hljs-comment"># 获取生产者生产的数据，如果队列中没有数据会阻塞，等待队列中有数据再获取</span>
        value = q.get()
        print(<span class="hljs-string">"消费者消费了%s数据"</span>%(value))
    print(<span class="hljs-string">"结束消费子进程……"</span>)

<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    q = Queue()

    p1 = Process(target=product, args=(q,))
    p2 = Process(target=customer, args=(q,))

    p1.start()
    p2.start()

    p1.join()
    <span class="hljs-comment"># p2子进程里面是死循环，无法等待它的结束</span>
    <span class="hljs-comment"># p2.join()</span>
    <span class="hljs-comment"># 强制结束子进程</span>
    p2.terminate()

    print(<span class="hljs-string">"主进程结束"</span>)

7、封装进程

zutuanxue_comProcess.py

<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process
<span class="hljs-keyword">import</span> time

<span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">zutuanxue_comProcess</span><span class="hljs-params">(Process)</span>:</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__init__</span><span class="hljs-params">(self, name, word)</span>:</span>
        super().__init__()
        self.name = name
        self.word = word

    <span class="hljs-comment"># 必须叫run，当对象调用start()方法时，默认调用run方法</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run</span><span class="hljs-params">(self)</span>:</span>
        <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">5</span>):
            print(<span class="hljs-string">"%s is a %s man"</span> % (self.name, self.word))
            time.sleep(<span class="hljs-number">1</span>)

kaige.Process.py

<span class="hljs-keyword">from</span> multiprocessing <span class="hljs-keyword">import</span> Process
<span class="hljs-keyword">import</span> time

<span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">KaigeProcess</span><span class="hljs-params">(Process)</span>:</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__init__</span><span class="hljs-params">(self, name)</span>:</span>
        super().__init__()
        self.name = name

    <span class="hljs-comment"># 必须叫run，当对象调用start()方法时，默认调用run方法</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">run</span><span class="hljs-params">(self)</span>:</span>
        <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">2</span>):
            print(<span class="hljs-string">"%s is a hansome man"</span> % (self.name))
            time.sleep(<span class="hljs-number">1</span>)

main.py

<span class="hljs-keyword">import</span> time
<span class="hljs-keyword">from</span> zutuanxue_comProcess <span class="hljs-keyword">import</span> Zutuanxue_comProcess
<span class="hljs-keyword">from</span> kaigeProcess <span class="hljs-keyword">import</span> KaigeProcess
<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    t1 = time.time()

    p = Zutuanxue_comProcess(<span class="hljs-string">"zutuanxue_com"</span>, <span class="hljs-string">"good"</span>)
    p.start()
    p.join()

    p1 = KaigeProcess(<span class="hljs-string">"kaige"</span>)
    p1.start()
    p1.join()

    t2 = time.time()
    print(<span class="hljs-string">"耗时：%.2f"</span>%(t2-t1))

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

如有侵权请联系 cloudcommunity@tencent.com 删除

python

编程算法

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体同步曝光计划，欢迎热爱写作的你一起参与！

python

编程算法

登录后参与评论

0 条评论

热度

Python进程管理

Python进程管理

一、多任务原理

二、进程

1、概念

2、使用进程

3、全局变量在多个子进程中不能共享

4、启动大量子进程

5、单进程与多进程复制文件对比

6、进程间通信

7、封装进程

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐