Loading [MathJax]/jax/output/CommonHTML/config.js
前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >专栏 >未对齐原始内存的加载和存储操作

未对齐原始内存的加载和存储操作

原创
作者头像
DerekYuYi
修改于 2022-11-08 08:34:15
修改于 2022-11-08 08:34:15
1.8K0
举报
文章被收录于专栏:Swift-开源分析Swift-开源分析

提议:SE-0349

swift 目前没有提供从任意字节源(如二进制文件)加载数据的明确方法,这些文件中可以存储数据而不考虑内存中的对齐。当前提议旨在纠正这种情况。

方法 UnsafeRawPointer.load<T>(fromByteOffset offset: Int, as type: T.Type) -> T要求self+offset处的地址正确对齐,才能用来访问类型T的实例。如果尝试使用指针和字节偏移量的组合,但没有对齐T,会导致运行时 crash。一般来说,保存到文件或网络流中的数据与内存中的数据流并不是遵守同样的限制,往往无法对齐。因此,当将数据从这些源(文件或网络流等)复制到内存时,Swift 用户经常会遇到内存对齐不匹配。

举个例子,给定任务数据流,其中 4 个字节值在字节偏移量 3 到 7 之间编码:

代码语言:Swift
AI代码解释
复制
let data = Data([0x0, 0x0, 0x0, 0xff, 0xff, 0xff, 0xff, 0x0])

为了把数据流中的所有0xff字节提取转为UInt32, 我们可以使用load(as:), 如下:

代码语言:Swift
AI代码解释
复制
let result = data.dropFirst(3).withUnsafeBytes { $0.load(as: UInt32.self) }

当你运行上述代码时,会发生运行时 crash。因为这种情况下,load方法要求基础指针已经正确进行内存对齐,才能访问UInt32。所以这里需要其他解决方案。比如下面列举一种解决方案:

代码语言:Swift
AI代码解释
复制
let result = data.dropFirst(3).withUnsafeBytes { buffer -> UInt32 in
  var storage = UInt32.zero
  withUnsafeMutableBytes(of: &storage) {
    $0.copyBytes(from: buffer.prefix(MemoryLayout<UInt32>.size))
  }
  return storage
}

虽然上述解决方案可以达到效果,但是这里存在2个问题。第一,这个解决方案的意图表现不是那么明显,我理解为嵌套过多。第二,上述解决方案使用了2次拷贝,而不是预期的单个拷贝:第一个拷贝到正确对齐的原始缓冲区,然后第二个拷贝到最后正确类型的变量。我们期望可以用一份拷贝完成这项工作。

改善任意内存对齐的加载操作,很重要的类型是它的值是可以进行逐位复制的类型,而不需要引用计数操作。这些类型通常被称为 "POD"(普通旧数据)或普通类型。我们建议将未对齐加载操作的使用限制到这些 POD 类型里。

解决方案

为了支持UnsafeRawPointer, UnsafeRawBufferPointer 以及他们的可变类型(mutable)的内存未对齐加载,我们提议新增 API UnsafeRawPointer.loadUnaligned(fromByteOffset:as:)。当然这些类型将会明确限制为 POD 类型。那么什么情况下加载非 POD 类型?只有当原始内存是另一个活跃对象时,且该对象的内存构造已经正确对齐。原来的 API(load)会继续支持这种情况。新的 API (loadUnaligned) 在返回值类型是 POD 类型时, 将会在 debug 模式下发生断言 (assert) ,中止运行。release 情况下面会讲到。

UnsafeMutableRawPointer.storeBytes(of:toByteOffset:) API 只对 POD 类型起作用,这点在文档已经注释标明。但是在运行时,该 API 会将内存地址存储强制转为与原始类型已经正确对齐的偏移量。这里我们建议删除该对齐限制,并强制执行文档中标明的 POD 限制。这样虽然文档已经更新,但 API 可以保持不变。

UnsafeRawBufferPointerUnsafeMutableRawBufferPointer 类型都会接受相关的修改。

详细设计

代码语言:Swift
AI代码解释
复制
extension UnsafeRawPointer {
  /// Returns a new instance of the given type, constructed from the raw memory
  /// at the specified offset.
  ///
  /// This function only supports loading trivial types,
  /// and will trap if this precondition is not met.
  /// A trivial type does not contain any reference-counted property
  /// within its in-memory representation.
  /// The memory at this pointer plus `offset` must be laid out
  /// identically to the in-memory representation of `T`.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// - Parameters:
  ///   - offset: The offset from this pointer, in bytes. `offset` must be
  ///     nonnegative. The default is zero.
  ///   - type: The type of the instance to create.
  /// - Returns: A new instance of type `T`, read from the raw bytes at
  ///   `offset`. The returned instance isn't associated
  ///   with the value in the range of memory referenced by this pointer.
  public func loadUnaligned<T>(fromByteOffset offset: Int = 0, as type: T.Type) -> T
}
代码语言:Swift
AI代码解释
复制
extension UnsafeMutableRawPointer {
  /// Returns a new instance of the given type, constructed from the raw memory
  /// at the specified offset.
  ///
  /// This function only supports loading trivial types,
  /// and will trap if this precondition is not met.
  /// A trivial type does not contain any reference-counted property
  /// within its in-memory representation.
  /// The memory at this pointer plus `offset` must be laid out
  /// identically to the in-memory representation of `T`.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// - Parameters:
  ///   - offset: The offset from this pointer, in bytes. `offset` must be
  ///     nonnegative. The default is zero.
  ///   - type: The type of the instance to create.
  /// - Returns: A new instance of type `T`, read from the raw bytes at
  ///   `offset`. The returned instance isn't associated
  ///   with the value in the range of memory referenced by this pointer.
  public func loadUnaligned<T>(fromByteOffset offset: Int = 0, as type: T.Type) -> T

  /// Stores the given value's bytes into raw memory at the specified offset.
  ///
  /// The type `T` to be stored must be a trivial type. The memory
  /// must also be uninitialized, initialized to `T`, or initialized to
  /// another trivial type that is layout compatible with `T`.
  ///
  /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is
  /// initialized to the raw bytes of `value`. If the memory is bound to a
  /// type `U` that is layout compatible with `T`, then it contains a value of
  /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the
  /// bound type of the memory.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// If you need to store a copy of a value of a type that isn't trivial into memory,
  /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know
  /// the type of value previously in memory and initialize or assign the
  /// memory. For example, to replace a value stored in a raw pointer `p`,
  /// where `U` is the current type and `T` is the new type, use a typed
  /// pointer to access and deinitialize the current value before initializing
  /// the memory with a new value.
  ///
  ///     let typedPointer = p.bindMemory(to: U.self, capacity: 1)
  ///     typedPointer.deinitialize(count: 1)
  ///     p.initializeMemory(as: T.self, repeating: newValue, count: 1)
  ///
  /// - Parameters:
  ///   - value: The value to store as raw bytes.
  ///   - offset: The offset from this pointer, in bytes. `offset` must be
  ///     nonnegative. The default is zero.
  ///   - type: The type of `value`.
  public func storeBytes<T>(of value: T, toByteOffset offset: Int = 0, as type: T.Type)
}

UnsafeRawBufferPointerUnsafeMutableRawBufferPointer 都有类似的 loadUnaligned 函数。它允许从缓冲区的任意偏移量做加载操作,并遵循BufferPointer类型的通用索引验证规则:在调试模式下编译客户端代码时,将检查索引,而在发布模式下编译客户代码时,则不检查索引。

代码语言:Swift
AI代码解释
复制
extension UnsafeMutableRawBufferPointer {
  /// Returns a new instance of the given type, constructed from the raw memory
  /// at the specified offset.
  ///
  /// This function only supports loading trivial types.
  /// A trivial type does not contain any reference-counted property
  /// within its in-memory stored representation.
  /// The memory at `offset` bytes into the buffer must be laid out
  /// identically to the in-memory representation of `T`.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// You can use this method to create new values from the buffer pointer's
  /// underlying bytes. The following example creates two new `Int32`
  /// instances from the memory referenced by the buffer pointer `someBytes`.
  /// The bytes for `a` are copied from the first four bytes of `someBytes`,
  /// and the bytes for `b` are copied from the next four bytes.
  ///
  ///     let a = someBytes.load(as: Int32.self)
  ///     let b = someBytes.load(fromByteOffset: 4, as: Int32.self)
  ///
  /// The memory to read for the new instance must not extend beyond the buffer
  /// pointer's memory region---that is, `offset + MemoryLayout<T>.size` must
  /// be less than or equal to the buffer pointer's `count`.
  ///
  /// - Parameters:
  ///   - offset: The offset, in bytes, into the buffer pointer's memory at
  ///     which to begin reading data for the new instance. The buffer pointer
  ///     plus `offset` must be properly aligned for accessing an instance of
  ///     type `T`. The default is zero.
  ///   - type: The type to use for the newly constructed instance. The memory
  ///     must be initialized to a value of a type that is layout compatible
  ///     with `type`.
  /// - Returns: A new instance of type `T`, copied from the buffer pointer's
  ///   memory.
  public func loadUnaligned<T>(fromByteOffset offset: Int = 0, as type: T.Type) -> T
}

此外,UnsafeMutableBufferPointer.storeBytes(of:toByteOffset) 方法将像它对应的UnsafeMutablePointer.storeBytes(of:toByteOffset) 方法一样发生更改,不再在运行时强制对齐内存。同样,索引验证行为没有改变:当客户端代码在调试模式(debug)下编译时,将检查索引,而当客户端代码以发布模式(release)编译时,则不检查索引。

代码语言:Swift
AI代码解释
复制
extension UnsafeMutableRawBufferPointer {
  /// Stores a value's bytes into the buffer pointer's raw memory at the
  /// specified byte offset.
  ///
  /// The type `T` to be stored must be a trivial type. The memory must also be
  /// uninitialized, initialized to `T`, or initialized to another trivial
  /// type that is layout compatible with `T`.
  ///
  /// The memory written to must not extend beyond the buffer pointer's memory
  /// region---that is, `offset + MemoryLayout<T>.size` must be less than or
  /// equal to the buffer pointer's `count`.
  ///
  /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is
  /// initialized to the raw bytes of `value`. If the memory is bound to a
  /// type `U` that is layout compatible with `T`, then it contains a value of
  /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the
  /// bound type of the memory.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// If you need to store a copy of a value of a type that isn't trivial into memory,
  /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know
  /// the type of value previously in memory and initialize or assign the memory.
  ///
  /// - Parameters:
  ///   - offset: The offset in bytes into the buffer pointer's memory to begin
  ///     reading data for the new instance. The buffer pointer plus `offset`
  ///     must be properly aligned for accessing an instance of type `T`. The
  ///     default is zero.
  ///   - type: The type to use for the newly constructed instance. The memory
  ///     must be initialized to a value of a type that is layout compatible
  ///     with `type`.
  public func storeBytes<T>(of value: T, toByteOffset offset: Int = 0, as: T.Type)
}

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

评论
登录后参与评论
暂无评论
推荐阅读
编辑精选文章
换一批
centos7 安装ElasticSearch 6
ElasticSearch是一个基于 Lucene 的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎,基于 RESTful web 接口。Elasticsearch 是用 Java 开发的,并作为 Apache 许可条款下的开放源码发布,是当前流行的企业级搜索引擎。设计用于云计算中,能够达到实时搜索,稳定,可靠,快速,安装使用方便。
崔笑颜
2020/06/08
5990
centos7 安装ElasticSearch 6
ElasticSearch 安装
Elasticsearch 的官方地址:https://www.elastic.co/cn/
用户9615083
2022/12/25
5860
ElasticSearch 安装
全文搜索引擎 Elasticsearch 入门:集群搭建
本文主要介绍什么是 ElasticSearch 以及为什么需要它,如何在本机安装部署 ElasticSearch 实例,同时会演示安装 ElasticSearch 插件,以及如何在本地部署多实例集群,方便在日后学习分布式相关原理。
武培轩
2020/02/28
7700
elasticsearch学习路程-安装篇
官网:https://www.elastic.co/cn/downloads/elasticsearch
吃完橙子了哈
2021/03/26
5180
linux elasticsearch-8.2.0安装
官网下载地址: https://www.elastic.co/cn/downloads/elasticsearch
oktokeep
2024/10/09
1340
linux elasticsearch-8.2.0安装
Java开发环境系列:全文搜索引擎elasticsearch(head插件)
https://github.com/mobz/elasticsearch-head
架构师小跟班
2019/08/06
4800
Java开发环境系列:全文搜索引擎elasticsearch(head插件)
es从线上库导出数据并导入开发环境
来了个需求,需要从某个线上es库查询一些数据出来并进行大屏展示。问需求方有没有开发环境的es库,答:没有,说要不直连他们的线上库。
低级知识传播者
2024/01/29
3690
es从线上库导出数据并导入开发环境
全文检索工具elasticsearch:第二章:安装配置
Elasticsearch官网: https://www.elastic.co/products/elasticsearch
Java廖志伟
2022/09/28
4300
全文检索工具elasticsearch:第二章:安装配置
【ES三周年】Linux安装ElasticSearch
Linux安装ElasticSearch 0. 写在前面 1. 环境准备 2. ES安装 2.1ES解压 2.2 新增普通用户 2.3 给新创建的普通用户授权 2.4 给新创建的普通用户设置sudo权限 2.5 前置准备 3. Kibana安装 3.1 上传并解压tar文件 4. ik分词器的安装 4.1 为什么要安装其他分词器 4.2下载地址 0. 写在前面 Linux版本:CentOS7.5 ES版本:ElasticSearch-7.8.0 1. 环境准备 运行ElasticSearch,需要安装并配置
WHYBIGDATA
2023/02/14
1.8K0
【ES三周年】Linux安装ElasticSearch
Linux安装全文搜索引擎Elasticsearch 6.2.4
Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口。
静谧星空TEL
2022/01/06
7090
Linux安装全文搜索引擎Elasticsearch 6.2.4
ElasticSearch 的安装以及设置ElasticSearch外网访问以及过程中遇到的问题
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
用户8346838
2021/03/11
4.1K0
ELKstack简介及环境部署
Logstash的运行依赖于Java运行环境, Logstash 1.5以上不低于java 7推荐使用最新版本的Java。由于只是运行Java程序,而不是开发,下载JRE即可。
程序员同行者
2018/09/27
5630
ELKstack简介及环境部署
Elasticsearch6.4安装
下载地址:https://www.elastic.co/cn/downloads 假设系统安装好了对应的jdk,且jdk的版本要高于8。下面是具体的安装步骤。
良辰美景TT
2018/12/13
9060
Java开发环境系列:全文搜索引擎elasticsearch(kibana插件)
Kibana是一个开源的分析和可视化平台,设计用于和Elasticsearch一起工作。
架构师小跟班
2019/08/06
6930
Java开发环境系列:全文搜索引擎elasticsearch(kibana插件)
Elasticsearch环境搭建
环境准备 Elasticsearch:7.9.3 JDK: 15.1(虽然ES自带的有JDK,但是还是建议使用自己安装的) Kibana:7.9.3(最好跟es对应) CentOS: 7(内存2GB,CPU两核心),三台虚拟机器(买不起云服务) elasticsearch-head(可选) Elasticsearch 对应的JDK版本可以在这查看https://www.elastic.co/cn/support/matrix#matrix_jvm ,7.9.3对应JDK已经支持到JDK15了,所以这里选择
用户1215919
2021/12/28
3320
Elasticsearch环境搭建
快速学习ES6-Elasticsearch介绍和安装
而商品的数量非常多,而且分类繁杂。如果能正确的显示出用户想要的商品,并进行合理的过滤,尽快促成交易,是搜索系统要研究的核心。
cwl_java
2020/02/11
6310
入门教程 | 从安装部署开始学习 Elasticsearch
Elasticsearch 是目前最流行的搜索引擎,学习 ES 的第一步便是安装部署。本文内容包含以下三部分:
大数据技术架构
2019/08/16
1.3K0
入门教程 | 从安装部署开始学习 Elasticsearch
Elasticsearch 在Linux下的安装部署和配置
https://mirrors.aliyun.com/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-2009.iso
授客
2024/11/21
4290
Elasticsearch-01CentOS7单节点部署ES5.6.16
April 11, 2019 ,ES 发布了7.0.0 版本。 ES是一个搜索引擎,基于Lucene,提供了基于RESTful 的web接口,能够达到实时,稳定,可靠,快速的搜索引擎。
小小工匠
2021/08/17
5980
ES7学习笔记(一)Elasticsearch的安装与启动
Elasticsearch是一个非常好用的搜索引擎,和Solr一样,他们都是基于倒排索引的。今天我们就看一看Elasticsearch如何进行安装。
小忽悠
2020/02/26
1.5K0
ES7学习笔记(一)Elasticsearch的安装与启动
推荐阅读
相关推荐
centos7 安装ElasticSearch 6
更多 >
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档