Loading [MathJax]/jax/output/CommonHTML/config.js
前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >专栏 >未对齐原始内存的加载和存储操作

未对齐原始内存的加载和存储操作

原创
作者头像
DerekYuYi
修改于 2022-11-08 08:34:15
修改于 2022-11-08 08:34:15
1.8K0
举报
文章被收录于专栏:Swift-开源分析Swift-开源分析

提议:SE-0349

swift 目前没有提供从任意字节源(如二进制文件)加载数据的明确方法,这些文件中可以存储数据而不考虑内存中的对齐。当前提议旨在纠正这种情况。

方法 UnsafeRawPointer.load<T>(fromByteOffset offset: Int, as type: T.Type) -> T要求self+offset处的地址正确对齐,才能用来访问类型T的实例。如果尝试使用指针和字节偏移量的组合,但没有对齐T,会导致运行时 crash。一般来说,保存到文件或网络流中的数据与内存中的数据流并不是遵守同样的限制,往往无法对齐。因此,当将数据从这些源(文件或网络流等)复制到内存时,Swift 用户经常会遇到内存对齐不匹配。

举个例子,给定任务数据流,其中 4 个字节值在字节偏移量 3 到 7 之间编码:

代码语言:Swift
AI代码解释
复制
let data = Data([0x0, 0x0, 0x0, 0xff, 0xff, 0xff, 0xff, 0x0])

为了把数据流中的所有0xff字节提取转为UInt32, 我们可以使用load(as:), 如下:

代码语言:Swift
AI代码解释
复制
let result = data.dropFirst(3).withUnsafeBytes { $0.load(as: UInt32.self) }

当你运行上述代码时,会发生运行时 crash。因为这种情况下,load方法要求基础指针已经正确进行内存对齐,才能访问UInt32。所以这里需要其他解决方案。比如下面列举一种解决方案:

代码语言:Swift
AI代码解释
复制
let result = data.dropFirst(3).withUnsafeBytes { buffer -> UInt32 in
  var storage = UInt32.zero
  withUnsafeMutableBytes(of: &storage) {
    $0.copyBytes(from: buffer.prefix(MemoryLayout<UInt32>.size))
  }
  return storage
}

虽然上述解决方案可以达到效果,但是这里存在2个问题。第一,这个解决方案的意图表现不是那么明显,我理解为嵌套过多。第二,上述解决方案使用了2次拷贝,而不是预期的单个拷贝:第一个拷贝到正确对齐的原始缓冲区,然后第二个拷贝到最后正确类型的变量。我们期望可以用一份拷贝完成这项工作。

改善任意内存对齐的加载操作,很重要的类型是它的值是可以进行逐位复制的类型,而不需要引用计数操作。这些类型通常被称为 "POD"(普通旧数据)或普通类型。我们建议将未对齐加载操作的使用限制到这些 POD 类型里。

解决方案

为了支持UnsafeRawPointer, UnsafeRawBufferPointer 以及他们的可变类型(mutable)的内存未对齐加载,我们提议新增 API UnsafeRawPointer.loadUnaligned(fromByteOffset:as:)。当然这些类型将会明确限制为 POD 类型。那么什么情况下加载非 POD 类型?只有当原始内存是另一个活跃对象时,且该对象的内存构造已经正确对齐。原来的 API(load)会继续支持这种情况。新的 API (loadUnaligned) 在返回值类型是 POD 类型时, 将会在 debug 模式下发生断言 (assert) ,中止运行。release 情况下面会讲到。

UnsafeMutableRawPointer.storeBytes(of:toByteOffset:) API 只对 POD 类型起作用,这点在文档已经注释标明。但是在运行时,该 API 会将内存地址存储强制转为与原始类型已经正确对齐的偏移量。这里我们建议删除该对齐限制,并强制执行文档中标明的 POD 限制。这样虽然文档已经更新,但 API 可以保持不变。

UnsafeRawBufferPointerUnsafeMutableRawBufferPointer 类型都会接受相关的修改。

详细设计

代码语言:Swift
AI代码解释
复制
extension UnsafeRawPointer {
  /// Returns a new instance of the given type, constructed from the raw memory
  /// at the specified offset.
  ///
  /// This function only supports loading trivial types,
  /// and will trap if this precondition is not met.
  /// A trivial type does not contain any reference-counted property
  /// within its in-memory representation.
  /// The memory at this pointer plus `offset` must be laid out
  /// identically to the in-memory representation of `T`.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// - Parameters:
  ///   - offset: The offset from this pointer, in bytes. `offset` must be
  ///     nonnegative. The default is zero.
  ///   - type: The type of the instance to create.
  /// - Returns: A new instance of type `T`, read from the raw bytes at
  ///   `offset`. The returned instance isn't associated
  ///   with the value in the range of memory referenced by this pointer.
  public func loadUnaligned<T>(fromByteOffset offset: Int = 0, as type: T.Type) -> T
}
代码语言:Swift
AI代码解释
复制
extension UnsafeMutableRawPointer {
  /// Returns a new instance of the given type, constructed from the raw memory
  /// at the specified offset.
  ///
  /// This function only supports loading trivial types,
  /// and will trap if this precondition is not met.
  /// A trivial type does not contain any reference-counted property
  /// within its in-memory representation.
  /// The memory at this pointer plus `offset` must be laid out
  /// identically to the in-memory representation of `T`.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// - Parameters:
  ///   - offset: The offset from this pointer, in bytes. `offset` must be
  ///     nonnegative. The default is zero.
  ///   - type: The type of the instance to create.
  /// - Returns: A new instance of type `T`, read from the raw bytes at
  ///   `offset`. The returned instance isn't associated
  ///   with the value in the range of memory referenced by this pointer.
  public func loadUnaligned<T>(fromByteOffset offset: Int = 0, as type: T.Type) -> T

  /// Stores the given value's bytes into raw memory at the specified offset.
  ///
  /// The type `T` to be stored must be a trivial type. The memory
  /// must also be uninitialized, initialized to `T`, or initialized to
  /// another trivial type that is layout compatible with `T`.
  ///
  /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is
  /// initialized to the raw bytes of `value`. If the memory is bound to a
  /// type `U` that is layout compatible with `T`, then it contains a value of
  /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the
  /// bound type of the memory.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// If you need to store a copy of a value of a type that isn't trivial into memory,
  /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know
  /// the type of value previously in memory and initialize or assign the
  /// memory. For example, to replace a value stored in a raw pointer `p`,
  /// where `U` is the current type and `T` is the new type, use a typed
  /// pointer to access and deinitialize the current value before initializing
  /// the memory with a new value.
  ///
  ///     let typedPointer = p.bindMemory(to: U.self, capacity: 1)
  ///     typedPointer.deinitialize(count: 1)
  ///     p.initializeMemory(as: T.self, repeating: newValue, count: 1)
  ///
  /// - Parameters:
  ///   - value: The value to store as raw bytes.
  ///   - offset: The offset from this pointer, in bytes. `offset` must be
  ///     nonnegative. The default is zero.
  ///   - type: The type of `value`.
  public func storeBytes<T>(of value: T, toByteOffset offset: Int = 0, as type: T.Type)
}

UnsafeRawBufferPointerUnsafeMutableRawBufferPointer 都有类似的 loadUnaligned 函数。它允许从缓冲区的任意偏移量做加载操作,并遵循BufferPointer类型的通用索引验证规则:在调试模式下编译客户端代码时,将检查索引,而在发布模式下编译客户代码时,则不检查索引。

代码语言:Swift
AI代码解释
复制
extension UnsafeMutableRawBufferPointer {
  /// Returns a new instance of the given type, constructed from the raw memory
  /// at the specified offset.
  ///
  /// This function only supports loading trivial types.
  /// A trivial type does not contain any reference-counted property
  /// within its in-memory stored representation.
  /// The memory at `offset` bytes into the buffer must be laid out
  /// identically to the in-memory representation of `T`.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// You can use this method to create new values from the buffer pointer's
  /// underlying bytes. The following example creates two new `Int32`
  /// instances from the memory referenced by the buffer pointer `someBytes`.
  /// The bytes for `a` are copied from the first four bytes of `someBytes`,
  /// and the bytes for `b` are copied from the next four bytes.
  ///
  ///     let a = someBytes.load(as: Int32.self)
  ///     let b = someBytes.load(fromByteOffset: 4, as: Int32.self)
  ///
  /// The memory to read for the new instance must not extend beyond the buffer
  /// pointer's memory region---that is, `offset + MemoryLayout<T>.size` must
  /// be less than or equal to the buffer pointer's `count`.
  ///
  /// - Parameters:
  ///   - offset: The offset, in bytes, into the buffer pointer's memory at
  ///     which to begin reading data for the new instance. The buffer pointer
  ///     plus `offset` must be properly aligned for accessing an instance of
  ///     type `T`. The default is zero.
  ///   - type: The type to use for the newly constructed instance. The memory
  ///     must be initialized to a value of a type that is layout compatible
  ///     with `type`.
  /// - Returns: A new instance of type `T`, copied from the buffer pointer's
  ///   memory.
  public func loadUnaligned<T>(fromByteOffset offset: Int = 0, as type: T.Type) -> T
}

此外,UnsafeMutableBufferPointer.storeBytes(of:toByteOffset) 方法将像它对应的UnsafeMutablePointer.storeBytes(of:toByteOffset) 方法一样发生更改,不再在运行时强制对齐内存。同样,索引验证行为没有改变:当客户端代码在调试模式(debug)下编译时,将检查索引,而当客户端代码以发布模式(release)编译时,则不检查索引。

代码语言:Swift
AI代码解释
复制
extension UnsafeMutableRawBufferPointer {
  /// Stores a value's bytes into the buffer pointer's raw memory at the
  /// specified byte offset.
  ///
  /// The type `T` to be stored must be a trivial type. The memory must also be
  /// uninitialized, initialized to `T`, or initialized to another trivial
  /// type that is layout compatible with `T`.
  ///
  /// The memory written to must not extend beyond the buffer pointer's memory
  /// region---that is, `offset + MemoryLayout<T>.size` must be less than or
  /// equal to the buffer pointer's `count`.
  ///
  /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is
  /// initialized to the raw bytes of `value`. If the memory is bound to a
  /// type `U` that is layout compatible with `T`, then it contains a value of
  /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the
  /// bound type of the memory.
  ///
  /// - Note: A trivial type can be copied with just a bit-for-bit copy without
  ///   any indirection or reference-counting operations. Generally, native
  ///   Swift types that do not contain strong or weak references or other
  ///   forms of indirection are trivial, as are imported C structs and enums.
  ///
  /// If you need to store a copy of a value of a type that isn't trivial into memory,
  /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know
  /// the type of value previously in memory and initialize or assign the memory.
  ///
  /// - Parameters:
  ///   - offset: The offset in bytes into the buffer pointer's memory to begin
  ///     reading data for the new instance. The buffer pointer plus `offset`
  ///     must be properly aligned for accessing an instance of type `T`. The
  ///     default is zero.
  ///   - type: The type to use for the newly constructed instance. The memory
  ///     must be initialized to a value of a type that is layout compatible
  ///     with `type`.
  public func storeBytes<T>(of value: T, toByteOffset offset: Int = 0, as: T.Type)
}

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

评论
登录后参与评论
暂无评论
推荐阅读
编辑精选文章
换一批
浅谈Golang内存对齐
如果你在 golang spec 里以「alignment」为关键字搜索的话,那么会发现与此相关的内容并不多,只是在结尾介绍 unsafe 包的时候提了一下,不过别忘了字儿越少事儿越大:
LA0WAN9
2021/12/14
1.4K0
浅谈Golang内存对齐
Swift 对象内存模型探究(一)
HandyJSON 是 Swift 处理 JSON 数据的开源库之一,类似 JOSNModel,它可以直接将 JSON 数据转化为类实例在代码中使用。 由于 Swift 是一种静态语言,没有 OC 那种灵活的 Runtime 机制,为了达到类似 JSONModel 的效果,HandyJSON 另辟蹊径,绕过对 Runtime 的依赖,直接操作实例的内存对实例属性进行赋值,从而得到一个完全初始化完成的实例。 本文将通过探究 Swift 对象内存模型机制,简单介绍 HandyJSON 实现原理. 内存分配 St
腾讯Bugly
2018/03/23
2.1K0
redis中ziplist
ziplist 是一个压缩的双向列表。传统的双向链表,在每个节点,都需要指向下一个和前一个节点的指针,占据了一定的空间;同时双向链表中使用字符串保存了节点的值,对于整形的数值而言,比较费空间。ziplist 在这些方面进行了一些优化。
全栈程序员站长
2022/07/25
2600
【redis源码学习】redis 专属“链表”:ziplist
用过 Python 的列表吗?就是那种可以存储任意类型数据的,支持随机读取的数据结构。 没有用过的话那就没办法了。
看、未来
2021/12/24
2640
【redis源码学习】redis 专属“链表”:ziplist
JVM源码实战 - OOP-Klass模型
OOP-Klass模型用来描述class的属性和行为 设计为OOP和Klass两部分是因为不希望每个对象都有一个C ++ vtbl指针, 因此,普通的oops没有任何虚拟功能。 相反,他们将所有“虚拟”函数转发到它们的klass,它具有vtbl并根据对象的实际类型执行C ++调度。
JavaEdge
2021/02/22
5270
JVM源码实战 -  OOP-Klass模型
golang源码分析:go-json(1)
https://github.com/goccy/go-json起步比较晚,但是它大量参考了json-iterator/go的思路,同时也进行来一系列优化。它具体做了哪些优化呢,首先看下序列化:
golangLeetcode
2023/09/06
2350
golang源码分析:go-json(1)
copy-and-swap idiom
This answer is from https://stackoverflow.com/a/3279550/10133369
努力努力再努力F
2019/12/12
9510
手摸手Go 你的内存对齐了吗?
谈到内存对齐,早年间玩Java的时候就能偶尔打打交道,为此Java8还提供了个语法糖@Contended来帮助我们解决高速缓存cacheline内存未对齐的伪共享问题。不过Go目前涉及到类似问题,比如内存对齐带来的原子操作的问题还是需要手动处理下,毕竟Russ Cox大佬也发话了
用户3904122
2022/06/29
5700
手摸手Go 你的内存对齐了吗?
【十分钟教会你汇编】MIPS编程入门
无意中找到一篇十分好用,而且篇幅也不是很大的入门教程,通篇阅后,再把“栗子”敲一遍,基本可以有一个比较理性的认识,从而方便更好地进一步深入学习。
云深无际
2020/10/14
2.8K1
使用按位运算符创建内存对齐的数据结构
内存对齐是计算机编程中的一个重要概念,它确保了高效的内存访问,并有可能在各种性能关键型系统和应用中产生可观的性能提升。
Michel_Rolle
2023/11/20
2.7K0
C++/CLI(二)Mono C++/CLI Native调用和P/Invoke调用
本文根据Mono C++原文档翻译,这篇文章的目的,就是想说CLR程序在VS下面生成的DLL不能给Unity调用,因为Mono的Native调用的编码和MS CLR的不一样,如果Unity想要去调用C++程序,需要使用P/Invoke的方式,这两者的不兼容使得本来非常方便的C++/CLI在Unity下毫无用武之地,希望有一天MS能够给Mono CLR一片土地,方便你我他,还有就是高高兴兴写了半个月MS CLR以为能在Unity下使用了,结果一Run就炸,所以说以后代码未动,单元测试一定要先写啊,这片区代码需要全部重构了,血与泪的教训。
Pulsar-V
2019/04/01
3.8K0
[golang][history]The Go Annotated Specification\ Go注释规范 266b9d49bfa3d2d16b4111378b1f9794373ee141
This document supersedes all previous Go spec attempts. The intent is to make this a reference for syntax and semantics. It is annotated with additional information not strictly belonging into a language spec.
landv
2021/01/29
6370
[Matlab]VS和Matlab混合编程(相关API使用-进阶)
原文链接:https://blog.csdn.net/humanking7/article/details/85939988
祥知道
2020/03/10
6380
DAY86:阅读Kernel Execution
我们正带领大家开始阅读英文的《CUDA C Programming Guide》,今天是第85天,我们正在讲解Driver API,希望在接下来的15天里,您可以学习到原汁原味的CUDA,同时能养成英文阅读的习惯。
GPUS Lady
2018/12/07
1K0
Go Quick Start 极简教程
IDE :使用 GoLand is a cross-platform IDE built specially for Go developers。 https://www.jetbrains.com/go/
一个会写诗的程序员
2021/05/06
7650
Win32k NULL-Pointer-Dereference Analysis by Matching the May Update
Microsoft shipped and fixed four win32k kernel Escalation of Privilege vulnerabilities in the May security bulletin. This article will discover and analyze one of these vulnerabilities caused by a null pointer dereference fixed by the patch program, and will finally attempt to implement its proof and exploitation code. The analyzing and debugging process will take place in a virtual machine of Windows 7 x86 SP1 basic environment.
稻草小刀
2022/12/12
4040
Win32k NULL-Pointer-Dereference Analysis by Matching the May Update
从Rust到远方:WebAssembly 星系
来源:https://mnt.io/2018/08/22/from-rust-to-beyond-the-webassembly-galaxy/
MikeLoveRust
2019/07/30
1.6K0
Swift 的 MemoryLayout 是如何工作的(1)
自从在 搜狐技术产品 公众号看过 一文看破Swift枚举本质 后,就一直计划在该文章的基础更加深入地挖掘一下 Swift 枚举的内存布局。但是,Swift 枚举的内存布局 涉及的内容比较多。所以,就先把 Swift 的 MemoryLayout 是如何工作的 部分拆出来单独写两篇文章。
酷酷的哀殿
2020/10/26
1.2K0
Swift 的 MemoryLayout 是如何工作的(1)
[golang][history]The Go Annotated Specification\ Go注释规范328df636c5f3e0875bc71a7eadf5a4a5084e0b13
This document supersedes all previous Go spec attempts. The intent is to make this a reference for syntax and semantics. It is annotated with additional information not strictly belonging into a language spec.
landv
2021/01/29
7360
Swift 中的反射 Mirror
前言 Mirror是Swift中的反射机制,对于C#和Java开发人员来说,应该很熟悉反射这个概念。反射就是可以动态的获取类型以及成员信息,同时也可以在运行时动态的调用方法和属性等。
Swift社区
2021/11/26
5.2K0
Swift 中的反射 Mirror
相关推荐
浅谈Golang内存对齐
更多 >
领券
💥开发者 MCP广场重磅上线!
精选全网热门MCP server,让你的AI更好用 🚀
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档