Android Codec2处理流程适配和解析

原创

colourfate

修改于 2024-03-08 10:01:20

1.2K0

修改于 2024-03-08 10:01:20

1 介绍

Codec2是Android中多媒体相关的软件框架，是MediaCodec的中间件，往上对接MediaCodec Native层，往下提供新的API标准供芯片底层的编解码去实现，也就是说适配了Codec2，就可以通过MediaCodec来调用芯片的硬件编解码的能力，来完成一些多媒体相关的功能。这篇文章先从下到上讲解适配Codec2需要实现的接口，然后再从上到下分析MediaCodec的流程来分析这些接口是如何调用的。主要抓住以下两条主线

输入buffer是如何送到编解码组件的
编解码完成之后输入buffer和输出buffer是如何上报的

开始之前需要如下前置知识

Android异步消息机制
Android HIDL
视频编解码基本流程

2 适配

下面以Android中的软件Hevc编码器的实现为例，分析如何适配Codec2接口，首先看codec2的基本架构，分为4层，第一层是sfplugin，负责和上层的stage fright对接，下面是HIDL，是各个组件的硬件抽象层，再往下是Core，封装了一个组件需要实现的接口，最后是具体的Component实现，这里以Hevc软编码器为例，再往下就是具体的编解码库了，软编码器调用的是ihevce相关的接口

2.1 目录结构

Android中的codec2目录在frameworks/av/media/codec2

.
├── Android.mk
├── components     # 适配的组件，如h264、hevc软件编解码等，由HIDL调用
├── core           # codec2内核，对接component
├── docs
├── faultinjection
├── fuzzer
├── hidl           # hal层实现
├── OWNERS
├── sfplugin       # 和stagefright的对接层
├── TEST_MAPPING
├── tests
└── vndk

core层组织了components的运行方式，这里先分析core层，其中主要的文件是：core/include/C2Component.h，其中包含了C2Component和C2ComponentInterface两个类

2.2 C2Component

C2Component中定义了一个组件需要实现的接口，定义如下，这里需要关注的两个重要接口

queue_nb：可以看作送帧/送流的接口，在编解码之前，将需要处理的原始数据送入，该接口必现设计为非阻塞的
onWorkDone_nb：当一帧数据处理完了之后会回调该接口

class C2Component {
public:
    class Listener {
    public:
        virtual void onWorkDone_nb(std::weak_ptr<C2Component> component,
                                std::list<std::unique_ptr<C2Work>> workItems) = 0;
        virtual void onTripped_nb(std::weak_ptr<C2Component> component,
                               std::vector<std::shared_ptr<C2SettingResult>> settingResult) = 0;
        virtual void onError_nb(std::weak_ptr<C2Component> component,
                             uint32_t errorCode) = 0;
        virtual ~Listener() = default;
    };

    ...
    /* Queues up work for the component. */
    virtual c2_status_t queue_nb(std::list<std::unique_ptr<C2Work>>* const items) = 0;
    /*
     * Announces a work to be queued later for the component. This reserves a slot for the queue
     * to ensure correct work ordering even if the work is queued later.
     */
    virtual c2_status_t announce_nb(const std::vector<C2WorkOutline> &items) = 0;

    enum flush_mode_t : uint32_t {
        /// flush work from this component only
        FLUSH_COMPONENT,

        /// flush work from this component and all components connected downstream from it via
        /// tunneling
        FLUSH_CHAIN = (1 << 16),
    };

    /*
     * Discards and abandons any pending work for the component, and optionally any component
     * downstream.
     */
    virtual c2_status_t flush_sm(flush_mode_t mode, std::list<std::unique_ptr<C2Work>>* const flushedWork) = 0;

    enum drain_mode_t : uint32_t {
        DRAIN_COMPONENT_WITH_EOS,
        DRAIN_COMPONENT_NO_EOS = (1 << 0),
        DRAIN_CHAIN = (1 << 16),
    };

    /*
     * Drains the component, and optionally downstream components. This is a signalling method;
     * as such it does not wait for any work completion.
     * Marks last work item as "drain-till-here", so component is notified not to wait for further
     * work before it processes work already queued. This method can also used to set the
     * end-of-stream flag after work has been queued. Client can continue to queue further work
     * immediately after this method returns.
     */
    virtual c2_status_t drain_nb(drain_mode_t mode) = 0;

    // STATE CHANGE METHODS
    // =============================================================================================
    virtual c2_status_t start() = 0;
    virtual c2_status_t stop() = 0;
    virtual c2_status_t reset() = 0;
    virtual c2_status_t release() = 0;
    virtual std::shared_ptr<C2ComponentInterface> intf() = 0;

    virtual ~C2Component() = default;
};

2.3 C2ComponentInterface

TODO

2.4 SimpleC2Component

SimpleC2Component提供了一种组件的实现，后面不同的实现只需要继承该实现即可，其中重要的接口设计如下。可以看到SimpleC2Component继承了C2Component，并将实现了一个AMessage和AHandle的异步消息机制，下面分别对相应的函数进行分析

class SimpleC2Component
        : public C2Component, public std::enable_shared_from_this<SimpleC2Component> {
public:
    explicit SimpleC2Component(
            const std::shared_ptr<C2ComponentInterface> &intf);
    virtual ~SimpleC2Component();

    // 设置对应的回调
    virtual c2_status_t setListener_vb(
            const std::shared_ptr<Listener> &listener, c2_blocking_t mayBlock) override;
    // 实现queue_nb接口
    virtual c2_status_t queue_nb(std::list<std::unique_ptr<C2Work>>* const items) override;
    ...
    // 实际处理一帧数据的函数
    bool processQueue();

protected:
    ...
    // 具体的编解码过程，由子类实现
    virtual void process(
            const std::unique_ptr<C2Work> &work,
            const std::shared_ptr<C2BlockPool> &pool) = 0;
    ...

    private:
    const std::shared_ptr<C2ComponentInterface> mIntf;

    class WorkHandler : public AHandler {
    public:
        ...

    protected:
        // 异步消息处理接口
        void onMessageReceived(const sp<AMessage> &msg) override;

    private:
        ...
    };
    ...
    struct ExecState {
        ExecState() : mState(UNINITIALIZED) {}

        int mState;
        std::shared_ptr<C2Component::Listener> mListener;
    };
    // 状态机
    Mutexed<ExecState> mExecState;

    sp<ALooper> mLooper;
    sp<WorkHandler> mHandler;

    class WorkQueue {
        ...
    }
    // 待处理的工作队列
    Mutexed<WorkQueue> mWorkQueue;
    ...

2.4.1 setListener_vb

该接口将上面的传递的listener设置到状态机中，后续时机合适时再回调相应接口

c2_status_t SimpleC2Component::setListener_vb(
        const std::shared_ptr<C2Component::Listener> &listener, c2_blocking_t mayBlock) {
    mHandler->setComponent(shared_from_this());

    Mutexed<ExecState>::Locked state(mExecState);
    ...
    state->mListener = listener;

    return C2_OK;
}

2.4.2 queue_nb

该函数实现了C2Component.queue_nb，实际只是将work放到队列中，并且发起一个异步消息然后返回，满足非阻塞的要求

c2_status_t SimpleC2Component::queue_nb(std::list<std::unique_ptr<C2Work>> * const items) {
    ...
    bool queueWasEmpty = false;
    {
        Mutexed<WorkQueue>::Locked queue(mWorkQueue);
        queueWasEmpty = queue->empty();
        while (!items->empty()) {
            queue->push_back(std::move(items->front()));
            items->pop_front();
        }
    }
    if (queueWasEmpty) {
        // 发起一次数据处理的消息，会将mWorkQueue中帧处理完
        (new AMessage(WorkHandler::kWhatProcess, mHandler))->post();
    }
    return C2_OK;
}

2.4.3 onMessageReceived

该函数是异步消息处理接口，当发出对应的消息时最终会调用到该函数中，这里processQueue()函数是处理一帧数据，然后返回当前队列是否还有未处理的数据

void SimpleC2Component::WorkHandler::onMessageReceived(const sp<AMessage> &msg) {
    ...

    switch (msg->what()) {
        case kWhatProcess: {
            if (mRunning) {
                // 如果processQueue()返回true，重新发一个kWhatProcess消息
                if (thiz->processQueue()) {
                    (new AMessage(kWhatProcess, this))->post();
                }
            } else {
                ALOGV("Ignore process message as we're not running");
            }
            break;
        }
        ...
    }
}

2.4.4 processQueue

该函数完成编解码的实际操作，代码如下，其中process和onWorkDone_nb都由子类实现，work先从之前的mWorkQueue队列中拿出，再调用process进行处理，process是一个虚函数，由子类实现，处理完成之后再调用listener->onWorkDone_nb通知处理完成事件onWorkDone_nb也是一个虚函数，由子类实现

bool SimpleC2Component::processQueue() {
    ...
    bool hasQueuedWork = false;

    {
        Mutexed<WorkQueue>::Locked queue(mWorkQueue);
        if (queue->empty()) {
            return false;
        }

        ...
        work = queue->pop_front();
        hasQueuedWork = !queue->empty();
    }

    if (!mOutputBlockPool) {
        c2_status_t err = [this] {
            ...
            std::shared_ptr<C2BlockPool> blockPool;
            err = GetCodec2BlockPool(poolId, shared_from_this(), &blockPool);
            ALOGD("Using output block pool with poolID %llu => got %llu - %d",
                    (unsigned long long)poolId,
                    (unsigned long long)(
                            blockPool ? blockPool->getLocalId() : 111000111),
                    err);
            if (err == C2_OK) {
                // mOutputBlockPool作用是什么？
                mOutputBlockPool = std::make_shared<BlockingBlockPool>(blockPool);
            }
            return err;
        }();
        ...
    }

    ...
    process(work, mOutputBlockPool);
    ...
    // 查看工作链中已处理完的数量，其实就是看proess是否处理成功？
    if (work->workletsProcessed != 0u) {
        queue.unlock();
        Mutexed<ExecState>::Locked state(mExecState);
        ALOGV("returning this work");
        std::shared_ptr<C2Component::Listener> listener = state->mListener;
        state.unlock();
        listener->onWorkDone_nb(shared_from_this(), vec(work));
    } else {
        ...
    }

    return hasQueuedWork;
}

2.5 C2SoftHevcEnc

下面分析process的实现，以Android Hevc软编码器为例，类继承自SimpleC2Component，实现如下，主要流程是从work中取出输入buffer，然后进行一帧编码，然后再把输出设置到work中

struct C2SoftHevcEnc : public SimpleC2Component {
    ...
    void process(const std::unique_ptr<C2Work>& work,
                 const std::shared_ptr<C2BlockPool>& pool) override;
    ...
}

void C2SoftHevcEnc::process(const std::unique_ptr<C2Work>& work,
                            const std::shared_ptr<C2BlockPool>& pool) {
    ...

    std::shared_ptr<const C2GraphicView> view;
    std::shared_ptr<C2Buffer> inputBuffer = nullptr;
    ...
    if (!work->input.buffers.empty()) {
        // 获取work中的input buffer
        inputBuffer = work->input.buffers[0];
        // 将buffer与view绑定
        view = std::make_shared<const C2GraphicView>(
            inputBuffer->data().graphicBlocks().front().map().get());
        ...
    }

    ...
    ihevce_inp_buf_t s_encode_ip{};
    ihevce_out_buf_t s_encode_op{};
    ...
    // 将view转换为s_encode_ip
    status = setEncodeArgs(&s_encode_ip, view.get(), workIndex);

    ...
    memset(&s_encode_op, 0, sizeof(s_encode_op));
    ...
    if (inputBuffer) {
        // 以s_encode_ip为输入，完成一帧hevc编码
        err = ihevce_encode(mCodecCtx, &s_encode_ip, &s_encode_op);
        ...
    }

    ...
    if (s_encode_op.i4_bytes_generated) {
        // s_encode_op有数据，将其配置到work中
        finishWork(s_encode_op.u8_pts, work, pool, &s_encode_op);
    }
}

下面分析finishWork，可见软编码输出的buffer最终是拷贝到了一个C2Buffer中，最终再放入work->worklets.front()->output.buffers队列

void C2SoftHevcEnc::finishWork(uint64_t index,
                               const std::unique_ptr<C2Work>& work,
                               const std::shared_ptr<C2BlockPool>& pool,
                               ihevce_out_buf_t* ps_encode_op) {
    std::shared_ptr<C2LinearBlock> block;
    ...
    // 获取一个LinearBlock
    c2_status_t status =
        pool->fetchLinearBlock(ps_encode_op->i4_bytes_generated, usage, &block);
    ...
    // 将block映射到view
    C2WriteView wView = block->map().get();
    ...
    // 将输出buffer内容拷贝到block中
    memcpy(wView.data(), ps_encode_op->pu1_output_buf,
           ps_encode_op->i4_bytes_generated);
    // 由block创建一个C2Buffer
    std::shared_ptr<C2Buffer> buffer =
        createLinearBuffer(block, 0, ps_encode_op->i4_bytes_generated);

    ...
    // 将buffer放到work中
    auto fillWork = [buffer](const std::unique_ptr<C2Work>& work) {
        work->worklets.front()->output.flags = (C2FrameData::flags_t)0;
        work->worklets.front()->output.buffers.clear();
        work->worklets.front()->output.buffers.push_back(buffer);
        work->worklets.front()->output.ordinal = work->input.ordinal;
        work->workletsProcessed = 1u;
    };

    if (work && c2_cntr64_t(index) == work->input.ordinal.frameIndex) {
        fillWork(work);
        ...
    } else {
       finish(index, fillWork);
    }
}

3 HIDL

3.1 概念

C2Component的上层是HIDL层，可以理解为Android的HAL层，这一层的头文件所继承的接口由一种叫做HIDL（Hardware Interface Definition Language）的语言动态生成，输出到out目录下，例如其中的IComponent头文件位于：

out/soong/.intermediates/hardware/interfaces/media/c2/1.0/android.hardware.media.c2@1.0_genc++_headers/gen/android/hardware/media/c2/1.0/IComponent.h

其定义如下

struct IComponent : public ::android::hidl::base::V1_0::IBase {
    typedef ::android::hardware::details::i_tag _hidl_tag;
    static const char* descriptor;

    virtual bool isRemote() const override { return false; }
    virtual ::android::hardware::Return<::android::hardware::media::c2::V1_0::Status> queue(const ::android::hardware::media::c2::V1_0::WorkBundle& workBundle) = 0;

    using flush_cb = std::function<void(::android::hardware::media::c2::V1_0::Status status, const ::android::hardware::media::c2::V1_0::WorkBundle& flushedWorkBundle)>;
    virtual ::android::hardware::Return<void> flush(flush_cb _hidl_cb) = 0;
    ...
}

对应的HIDL文件为hardware/interfaces/media/c2/1.0/IComponent.hal

interface IComponent {
queue(WorkBundle workBundle) generates (Status status);
flush(
    ) generates (
        Status status,
        WorkBundle flushedWorkBundle
    );
...
}

3.2 Component

Component继承自IComponent.h，也就会实现其中的接口，通过Component就可以调用上面讲到的C2Component，其定义如下，我们重点看下其中的queue和Listener。

struct Component : public IComponent,
                   public std::enable_shared_from_this<Component> {
    ...
    virtual Return<Status> queue(const WorkBundle& workBundle) override;
    ...
protected:
    ...
    struct Listener;
    ...
}

3.2.1 queue

queue的实现如下，这里的mComponent实际上就是C2Component，这里如何实现的暂且不表，后面再分析，以软编码为例，因此这里最终调用的queue_nb实际上调用的是SimpleC2Component.queue_nb

Return<Status> Component::queue(const WorkBundle& workBundle) {
    std::list<std::unique_ptr<C2Work>> c2works;

    if (!objcpy(&c2works, workBundle)) {
        return Status::CORRUPTED;
    }

    // Register input buffers.
    for (const std::unique_ptr<C2Work>& work : c2works) {
        if (work) {
            InputBufferManager::
                    registerFrameData(mListener, work->input);
        }
    }

    /* 调用C2Component.queue_nb，如果组件继承自SimpleC2Component，
     * 则调用SimpleC2Component.queue_nb
     * 注意这里是非阻塞的
     */
    return static_cast<Status>(mComponent->queue_nb(&c2works));
}

3.2.1 Listener

再看Listener的定义，这里实际继承的是C2Component::Listener，并且对onWorkDone_nb进行了实现，因为onWorkDone_nb是回调函数，因此由调用者实现也是符合预期的。这里其实就是调用了另一个回调listener->onWorkDone

struct Component::Listener : public C2Component::Listener {
    Listener(const sp<Component>& component) :
        mComponent(component),
        mListener(component->mListener) {
    }

    ...
    virtual void onWorkDone_nb(
            std::weak_ptr<C2Component> /* c2component */,
            std::list<std::unique_ptr<C2Work>> c2workItems) override {
        ...

        sp<IComponentListener> listener = mListener.promote();
        if (listener) {
            WorkBundle workBundle;

            // 拷贝到workBundle
            sp<Component> strongComponent = mComponent.promote();
            beginTransferBufferQueueBlocks(c2workItems, true);
            if (!objcpy(&workBundle, c2workItems, strongComponent ?
                    &strongComponent->mBufferPoolSender : nullptr)) {
                ...
            }
            // 回调
            Return<void> transStatus = listener->onWorkDone(workBundle);
            ...
            endTransferBufferQueueBlocks(c2workItems, true, true);
        }
    }

以上的Listener只是一个定义，还要看该Listener是在哪里声明的，以及是什么时候注册的。首先第一个问题，Listener声明是在SimpleC2Component::ExecState.mListener，只要继承了SimpleC2Component内部就有该成员，第二个问题，注册是在Component::initListener函数中，定义如下，同样以软编码为例，这里mComponent->setListener_vb调用实际是SimpleC2Component.setListener_vb

void Component::initListener(const sp<Component>& self) {
    std::shared_ptr<C2Component::Listener> c2listener =
            std::make_shared<Listener>(self);
    /* 调用C2Component.setListener_vb，如果组件继承自SimpleC2Component，
     * 那么调用SimpleC2Component.setListener_vb
     */
    c2_status_t res = mComponent->setListener_vb(c2listener, C2_DONT_BLOCK);
    ...
}

关于HIDL还有很长的一个调用流程，这里暂且分析到这里，后续再从MediaCodec从上往下分析，看如何调用到HIDL的

4 MediaCodec

MediaCodec是Android app层来进行多媒体编解码的模块，分为java层和cpp层，这里只从cpp层切入

4.1 调用流程

首先来看MediaCodec是如何使用的，由于MediaCodec也基于AMessage机制，因此先要创建一个ALooper，然后传递到MediaCodec中，下面以创建Hevc编码器为例，伪代码如下

sp<android::ALooper> looper = new android::ALooper;    // 创建ALooper
looper->setName("TestLooper");
looper->start();

// 创建编码器，自动查找合适的编码器组件
mMediaCodec = MediaCodec::CreateByType(looper, "video/hevc", true);
mMediaCodec->start();

for (uint32_t i = 0; i < 100; i++) {
    // 从MediaCodec取一个输入buffer的index
    mMediaCodec->dequeueInputBuffer(&inputBufIdx, sTimeOut);
    // 获取输入buffer
    sp<MediaCodecBuffer> inputBuf;
    mMediaCodec->getInputBuffer(inputBufIdx, &inputBuf);
    // 写YUV到输入buffer
    int readSize = writeYUVToBuffer(inputBuf->data(), inputBuf->size());
    uint32_t flags = 0;
    if (readSize == 0) {
        flags  = BUFFER_FLAG_END_OF_STREAM;
    }
    // 将输入buffer送回MediaCodec
    mMediaCodec->queueInputBuffer(inputBufIdx, 0, readSize, getCurTimeUs(), flags);

    // 从MediaCodec取一个输出buffer的index
    size_t outputBufIdx, outputOffset, outputSize;
    mMediaCodec->dequeueOutputBuffer(&outputBufIdx, &outputOffset, &outputSize,
                            &outputPts, &outputFlags, sTimeOut);
    // 获取输出buffer
    sp<MediaCodecBuffer> outputBuf;
    mMediaCodec->getOutputBuffer(outputBufIdx, &outputBuf);
    // 将码流写到文件
    writeStreamToFile(outputBuf);
    // 向MediaCodec释放输出buffer
    mMediaCodec->releaseOutputBuffer(outputBufIdx)
}

4.2 发送输入buffer流程

4.2.1 queueInputBuffer

首先分析queueInputBuffer，看YUV是如何送到具体的编码器组件的，可以看到实际这里只是发了一个异步消息，将index送进去，然后调用PostAndAwaitResponse阻塞等待消息响应

status_t MediaCodec::queueInputBuffer(
        size_t index,
        size_t offset,
        size_t size,
        int64_t presentationTimeUs,
        uint32_t flags,
        AString *errorDetailMsg) {
    if (errorDetailMsg != NULL) {
        errorDetailMsg->clear();
    }

    sp<AMessage> msg = new AMessage(kWhatQueueInputBuffer, this);
    msg->setSize("index", index);
    msg->setSize("offset", offset);
    msg->setSize("size", size);
    msg->setInt64("timeUs", presentationTimeUs);
    msg->setInt32("flags", flags);
    msg->setPointer("errorDetailMsg", errorDetailMsg);

    sp<AMessage> response;
    // 注意这里是阻塞等待
    return PostAndAwaitResponse(msg, &response);
}

4.2.2 QueueInputBuffer消息处理

再查看MediaCodec的消息处理函数，由于queueInputBuffer是阻塞等待的，因此这里要调用PostReplyWithError之后，queueInputBuffer才返回，这里往后是调用了onQueueInputBuffer函数

void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
    switch (msg->what()) {
    ...
    case kWhatQueueInputBuffer:
        {
            sp<AReplyToken> replyID;
            CHECK(msg->senderAwaitsResponse(&replyID));

            ...

            status_t err = UNKNOWN_ERROR;
            if (!mLeftover.empty()) {
                mLeftover.push_back(msg);
                size_t index;
                msg->findSize("index", &index);
                err = handleLeftover(index);
            } else {
                err = onQueueInputBuffer(msg);
            }

            // 消息响应
            PostReplyWithError(replyID, err);
            break;
        }
    ...
    }
}

下面分析onQueueInputBuffer，可以看到这里实际是通过index获取到MediaCodecBuffer，并且送到了mBufferChannel->queueInputBuffer中，注意执行到此处最外层的queueInputBuffer仍然在等待消息响应，因此到这里为止都是阻塞的

struct MediaCodec : public AHandler {
private:
    ...
    // 两个port，输入是0，输出是1
    std::vector<BufferInfo> mPortBuffers[2];
    ...
    std::shared_ptr<BufferChannelBase> mBufferChannel;
}

status_t MediaCodec::onQueueInputBuffer(const sp<AMessage> &msg) {
    size_t index;
    ...

    CHECK(msg->findSize("index", &index));
    ...
    BufferInfo *info = &mPortBuffers[kPortIndexInput][index];
    sp<MediaCodecBuffer> buffer = info->mData;
    ...
    if (hasCryptoOrDescrambler() && !c2Buffer && !memory) {
        // 安全编码流程
        ...
    } else {
        // 非安编码流程
        mBufferChannel->queueInputBuffer(buffer);
    }
    ...
}

4.3 获取输入buffer流程

4.3.1 dequeueInputBuffer

下面看dequeueInputBuffer函数，该函数是获取一个空闲的输入buffer，这里是否空闲仍然需要底层的组件来通知，因此需要分析这里的向上通知的流程。可以看到该函数仍然是发起一个异步消息，然后阻塞等待响应

status_t MediaCodec::dequeueInputBuffer(size_t *index, int64_t timeoutUs) {
    sp<AMessage> msg = new AMessage(kWhatDequeueInputBuffer, this);
    msg->setInt64("timeoutUs", timeoutUs);

    sp<AMessage> response;
    status_t err;
    if ((err = PostAndAwaitResponse(msg, &response)) != OK) {
        return err;
    }

    CHECK(response->findSize("index", index));

    return OK;
}

4.3.2 kWhatDequeueInputBuffer消息处理

再回到onMessageReceived函数，最终会调用到dequeuePortBuffer函数，可以看到MediaCodec中有一个mAvailPortBuffers链表，存储着当前可用的buffer的index，当dequeue的时候只需要从这个链表中拿出第一个index就行了

void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
    switch (msg->what()) {
    ...
    case kWhatDequeueInputBuffer:
    {
        sp<AReplyToken> replyID;
        CHECK(msg->senderAwaitsResponse(&replyID));

        handleDequeueInputBuffer(replyID, true /* new request */);
        ...
        break;
    }
    ...
    }
}

bool MediaCodec::handleDequeueInputBuffer(const sp<AReplyToken> &replyID, bool newRequest) {
    ...

    ssize_t index = dequeuePortBuffer(kPortIndexInput);
    ...

    sp<AMessage> response = new AMessage;
    response->setSize("index", index);
    // 响应消息
    response->postReply(replyID);

    return true;
}

struct MediaCodec : public AHandler {
private:
    ...
    // 存储可用的index
    List<size_t> mAvailPortBuffers[2];
    ...
}

ssize_t MediaCodec::dequeuePortBuffer(int32_t portIndex) {
    ...
    List<size_t> *availBuffers = &mAvailPortBuffers[portIndex];
    size_t index = *availBuffers->begin();
    ...
    availBuffers->erase(availBuffers->begin());

    return index;
}

4.3.3 updateBuffers

继续分析availBuffers是什么时候更新的，查看updateBuffers函数，发现其中的index是通过异步消息上报的，updateBuffers可以更新输入队列和输出队列，我们只看输入队列，发现是在收到kWhatFillThisBuffer消息时更新的

size_t MediaCodec::updateBuffers(
        int32_t portIndex, const sp<AMessage> &msg) {
    CHECK(portIndex == kPortIndexInput || portIndex == kPortIndexOutput);
    size_t index;
    // 从消息中获取index
    CHECK(msg->findSize("index", &index));
    sp<RefBase> obj;
    CHECK(msg->findObject("buffer", &obj));
    // 获取buffer的实体
    sp<MediaCodecBuffer> buffer = static_cast<MediaCodecBuffer *>(obj.get());

    {
        Mutex::Autolock al(mBufferLock);
        // 如果index大于了队列的大小，则扩充队列
        if (mPortBuffers[portIndex].size() <= index) {
            mPortBuffers[portIndex].resize(align(index + 1, kNumBuffersAlign));
        }
        // 将buffer放到队列中，索引为index
        mPortBuffers[portIndex][index].mData = buffer;
    }
    // 将index放入可用的链表
    mAvailPortBuffers[portIndex].push_back(index);

    return index;
}

void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
    switch (msg->what()) {
    ...
    case kWhatFillThisBuffer:
    {
        updateBuffers(kPortIndexInput, msg);
        ...
        break;
    }
    ...
}

4.3.4 BufferCallback

至此可以发现，MediaCodec中的输入和输出buffer是否可用时由异步消息通知的，而异步消息又是底层的组件通过回调MediaCodec的接口发送的，这部分代码在BufferCallback中，实际上BufferCallback是继承了CodecBase::BufferCallback类，并实现了其中的接口，可以看到输入和输出buffer都是在这里回调的

class BufferCallback : public CodecBase::BufferCallback {
public:
    explicit BufferCallback(const sp<AMessage> &notify);
    virtual ~BufferCallback() = default;

    virtual void onInputBufferAvailable(
            size_t index, const sp<MediaCodecBuffer> &buffer) override;
    virtual void onOutputBufferAvailable(
            size_t index, const sp<MediaCodecBuffer> &buffer) override;
private:
    const sp<AMessage> mNotify;
};

BufferCallback::BufferCallback(const sp<AMessage> &notify)
    : mNotify(notify) {}

void BufferCallback::onInputBufferAvailable(
        size_t index, const sp<MediaCodecBuffer> &buffer) {
    sp<AMessage> notify(mNotify->dup());
    notify->setInt32("what", kWhatFillThisBuffer);
    notify->setSize("index", index);
    notify->setObject("buffer", buffer);
    notify->post();
}

CodecBase实际上又是codec2中的接口了，属于sfplugin模块，篇幅原因这里暂时不往下分析了

4.4 获取输出buffer流程

4.4.1 dequeueOutputBuffer

获取输出buffer和获取输入buffer是一个流程，都是发出一个消息然后等待响应，这里输出buffer是底层组件处理好的数据，因此也需要底层组件来通知

status_t MediaCodec::dequeueOutputBuffer(
        size_t *index,
        size_t *offset,
        size_t *size,
        int64_t *presentationTimeUs,
        uint32_t *flags,
        int64_t timeoutUs) {
    sp<AMessage> msg = new AMessage(kWhatDequeueOutputBuffer, this);
    msg->setInt64("timeoutUs", timeoutUs);

    sp<AMessage> response;
    status_t err;
    if ((err = PostAndAwaitResponse(msg, &response)) != OK) {
        return err;
    }

    CHECK(response->findSize("index", index));
    CHECK(response->findSize("offset", offset));
    CHECK(response->findSize("size", size));
    CHECK(response->findInt64("timeUs", presentationTimeUs));
    CHECK(response->findInt32("flags", (int32_t *)flags));

    return OK;
}

4.4.2 kWhatDequeueOutputBuffer消息处理

获取输出buffer和获取输入buffer相同，都是调用的dequeuePortBuffer函数，只不过传入参数不同，也就是这里仍然是从mAvailPortBuffers队列中获取以及处理完成的buffer index

void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
    switch (msg->what()) {
    ...
    case kWhatDequeueOutputBuffer:
    {
        sp<AReplyToken> replyID;
        CHECK(msg->senderAwaitsResponse(&replyID));
        ...
        handleDequeueOutputBuffer(replyID, true /* new request */))
        ...
        break;
    }
    ...
    }
}

bool MediaCodec::handleDequeueOutputBuffer(const sp<AReplyToken> &replyID, bool newRequest) {
    ...
    } else {
        sp<AMessage> response = new AMessage;
        ...
        dequeuePortBuffer(kPortIndexOutput);
        ...
        response->postReply(replyID);
    }
}

4.4.3 mAvailPortBuffers的更新

获取输入buffer仍然是调用updateBuffers进行mAvailPortBuffers的更新，只不过这里是从kWhatDrainThisBuffer更新的，而发出该消息的地方也是另一个onOutputBufferAvailable回调

void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
    switch (msg->what()) {
    ...
    case kWhatDrainThisBuffer:
    {
        ...
        updateBuffers(kPortIndexInput, msg);
        ...
        break;
    }
    ...
}

void BufferCallback::onOutputBufferAvailable(
        size_t index, const sp<MediaCodecBuffer> &buffer) {
    sp<AMessage> notify(mNotify->dup());
    notify->setInt32("what", kWhatDrainThisBuffer);
    notify->setSize("index", index);
    notify->setObject("buffer", buffer);
    notify->post();
}

4.5 释放输出buffer

4.5.1 releaseOutputBuffer

释放输出buffer流程比较简单，也是发一个异步消息然后阻塞等待，在消息处理中将mPortBuffers队列对应的buffer清除，然后最后调用mBufferChannel->discardBuffer通知底层组件

status_t MediaCodec::releaseOutputBuffer(size_t index) {
    sp<AMessage> msg = new AMessage(kWhatReleaseOutputBuffer, this);
    msg->setSize("index", index);

    sp<AMessage> response;
    return PostAndAwaitResponse(msg, &response);
}

void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
    switch (msg->what()) {
    ...
    case kWhatDrainThisBuffer:
    {   
        ...
        onReleaseOutputBuffer(msg);
        PostReplyWithError(replyID, err);
        break;
    }
    ...
}

status_t MediaCodec::onReleaseOutputBuffer(const sp<AMessage> &msg) {
    size_t index;
    CHECK(msg->findSize("index", &index));

    ...
    BufferInfo *info = &mPortBuffers[kPortIndexOutput][index];
    ...
    sp<MediaCodecBuffer> buffer;
    {
        Mutex::Autolock al(mBufferLock);
        info->mOwnedByClient = false;
        buffer = info->mData;
        info->mData.clear();
    }

    if (render && buffer->size() != 0) {
        ...
    } else {
        mBufferChannel->discardBuffer(buffer);
    }
}