Elasticsearch使用：Vector API
原创
Elasticsearch使用：Vector API
原创
HLee
修改于 2021-01-18 02:23:33
修改于 2021-01-18 02:23:33
代码可运行
运行总次数：0
代码可运行
简介
目前这个功能还是处于试验阶段。在未来的版本中可能会出现变化。基于 Vector（向量）的打分目前分为一下两种：
它们都是基于功能进行评分的。在实际的使用中，我们必须注意的是：向量函数的计算过程中，所有匹配的文档均被线性扫描。 因此，期望查询时间随匹配文档的数量线性增长。 因此，我们建议使用查询参数限制匹配文档的数量。
Vector
我们首先创建一个叫做 books 的索引，并定义它的 mapping 如下：
PUT books
{
"mappings": {
"properties": {
"author": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"category": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"format": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"isbn13": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"pages": {
"type": "long"
},
"price": {
"type": "float"
},
"publisher": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"rating": {
"type": "float"
},
"release_year": {
"type": "date",
"format": "strict_year"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"vector_recommendation": {
"type": "dense_vector",
"dims": 3
}
}
}
}然后，我们使用 bulk API 接口来导入数据：
PUT books/_bulk
{ "index" : { "_id" : "database-internals" } }
{"isbn13":"978-1492040347","author":"Alexander Petrov", "title":"Database Internals: A deep-dive into how distributed data systems work","publisher":"O'Reilly","category":["databases","information systems"],"pages":350,"price":47.28,"format":"paperback","rating":4.5, "release_year" : "2019", "vector_recommendation" : [3.5, 4.5, 5.2]}
{ "index" : { "_id" : "designing-data-intensive-applications" } }
{"isbn13":"978-1449373320", "author":"Martin Kleppmann", "title":"Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems","publisher":"O'Reilly","category":["databases" ],"pages":590,"price":31.06,"format":"paperback","rating":4.4, "release_year" : "2017", "vector_recommendation" : [5.9, 4.4, 6.8]}
{ "index" : { "_id" : "kafka-the-definitive-guide" } }
{"isbn13":"978-1491936160","author":[ "Neha Narkhede", "Gwen Shapira", "Todd Palino"], "title":"Kafka: The Definitive Guide: Real-time data and stream processing at scale", "publisher":"O'Reilly","category":["databases" ],"pages":297,"price":37.31,"format":"paperback","rating":3.9, "release_year" : "2017", "vector_recommendation" : [2.97, 3.9, 6.2]}
{ "index" : { "_id" : "effective-java" } }
{"isbn13":"978-1491936160","author": "Joshua Block", "title":"Effective Java", "publisher":"Addison-Wesley", "category":["programming languages", "java" ],"pages":412,"price":27.91,"format":"paperback","rating":4.2, "release_year" : "2017", "vector_recommendation" : [4.12, 4.2, 7.2]}
{ "index" : { "_id" : "daemon" } }
{"isbn13":"978-1847249616","author":"Daniel Suarez", "title":"Daemon","publisher":"Quercus","category":["dystopia","novel"],"pages":448,"price":12.03,"format":"paperback","rating":4.0, "release_year" : "2011", "vector_recommendation" : [4.48, 4.0, 8.7]}
{ "index" : { "_id" : "cryptonomicon" } }
{"isbn13":"978-1847249616","author":"Neal Stephenson", "title":"Cryptonomicon","publisher":"Avon","category":["thriller", "novel" ],"pages":1152,"price":6.99,"format":"paperback","rating":4.0, "release_year" : "2002", "vector_recommendation" : [10.0, 4.0, 9.3]}
{ "index" : { "_id" : "garbage-collection-handbook" } }
{"isbn13":"978-1420082791","author": [ "Richard Jones", "Antony Hosking", "Eliot Moss" ], "title":"The Garbage Collection Handbook: The Art of Automatic Memory Management","publisher":"Taylor & Francis","category":["programming algorithms" ],"pages":511,"price":87.85,"format":"paperback","rating":5.0, "release_year" : "2011", "vector_recommendation" : [5.1, 5.0, 1.3] }
{ "index" : { "_id" : "radical-candor" } }
{"isbn13":"978-1250258403","author": "Kim Scott", "title":"Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity","publisher":"Macmillan","category":["human resources","management", "new work"],"pages":404,"price":7.29,"format":"paperback","rating":4.0, "release_year" : "2018", "vector_recommendation" : [4.0, 4.0, 9.2] }这样我们的索引 books 中有8个文档。我们仔细地查看一下我们输入的数据，它里面含有一个叫做 vecto_recommendation 的字段
"vector_recommendation" : [3.5, 4.5, 5.2]- 向量里的第一个数据3.5，实际上是我们的在这个文档里的 pages 除以100而得到的。如果这本书的页数越多，则表示这个数值越大。它的范围在0-10之间
- 向量里的第二个数据是这本书的 rating，也即评价。这个值在这个文档里的 rating 项可以查到。范围在0-5之间
- 向量里的第三个数据是书的价钱，这个值越低表明，价钱越贵，因为我们都喜欢便宜一点的书籍。0代表100元以上，10则表示10元以下的书
在上面的 mapping 中，我们是这样定义的：
"vector_recommendation": { "type": "dense_vector", "dims": 3 }它定义了这个 vector_recommendation 的类型是 dense_vector，它是一个3维的向量。
现在我们的数据都已经准备好了。我们接下来做一些我们喜欢的搜索。
vector搜索
在上面我们已经建立了我们的向量模型。那么我们怎么能够找到那些书的页数比较少，便宜的而且评价非常高的书呢？我们可以采用如下的搜索方式：
GET books/_search
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "cosineSimilarity(params.query_vector, doc['vector_recommendation']) + 1.0",
"params": {
"query_vector": [
1,
5,
10
]
}
}
}
}
}在上面的搜索中，我们通过脚本：
cosineSimilarity(params.query_vector, doc['vector_recommendation']) + 1.0来计算我们的搜索的分数。这里加上1的作用是为了避免我们最后的分数是负数。
在上面的表达式中：
"params": {"query_vector": [ 1, 5, 10 ] }我们想寻找的书是最好是100页的书，因为第一项是1；我们也同时想找一个评价好的书，因为第二项是5；同时我们想找最便宜的书，因为第三项是10。按照上面的要求，我们可以得到如下的搜索结果：
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 8,
"relation" : "eq"
},
"max_score" : 1.9568613,
"hits" : [
{
"_index" : "books",
"_type" : "_doc",
"_id" : "radical-candor",
"_score" : 1.9568613,
"_source" : {
"isbn13" : "978-1250258403",
"author" : "Kim Scott",
"title" : "Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity",
"publisher" : "Macmillan",
"category" : [
"human resources",
"management",
"new work"
],
"pages" : 404,
"price" : 7.29,
"format" : "paperback",
"rating" : 4.0,
"release_year" : "2018",
"vector_recommendation" : [
4.0,
4.0,
9.2
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "kafka-the-definitive-guide",
"_score" : 1.9520907,
"_source" : {
"isbn13" : "978-1491936160",
"author" : [
"Neha Narkhede",
"Gwen Shapira",
"Todd Palino"
],
"title" : "Kafka: The Definitive Guide: Real-time data and stream processing at scale",
"publisher" : "O'Reilly",
"category" : [
"databases"
],
"pages" : 297,
"price" : 37.31,
"format" : "paperback",
"rating" : 3.9,
"release_year" : "2017",
"vector_recommendation" : [
2.97,
3.9,
6.2
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "daemon",
"_score" : 1.9394372,
"_source" : {
"isbn13" : "978-1847249616",
"author" : "Daniel Suarez",
"title" : "Daemon",
"publisher" : "Quercus",
"category" : [
"dystopia",
"novel"
],
"pages" : 448,
"price" : 12.03,
"format" : "paperback",
"rating" : 4.0,
"release_year" : "2011",
"vector_recommendation" : [
4.48,
4.0,
8.7
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "effective-java",
"_score" : 1.9305289,
"_source" : {
"isbn13" : "978-1491936160",
"author" : "Joshua Block",
"title" : "Effective Java",
"publisher" : "Addison-Wesley",
"category" : [
"programming languages",
"java"
],
"pages" : 412,
"price" : 27.91,
"format" : "paperback",
"rating" : 4.2,
"release_year" : "2017",
"vector_recommendation" : [
4.12,
4.2,
7.2
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "database-internals",
"_score" : 1.9005439,
"_source" : {
"isbn13" : "978-1492040347",
"author" : "Alexander Petrov",
"title" : "Database Internals: A deep-dive into how distributed data systems work",
"publisher" : "O'Reilly",
"category" : [
"databases",
"information systems"
],
"pages" : 350,
"price" : 47.28,
"format" : "paperback",
"rating" : 4.5,
"release_year" : "2019",
"vector_recommendation" : [
3.5,
4.5,
5.2
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "designing-data-intensive-applications",
"_score" : 1.8525991,
"_source" : {
"isbn13" : "978-1449373320",
"author" : "Martin Kleppmann",
"title" : "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems",
"publisher" : "O'Reilly",
"category" : [
"databases"
],
"pages" : 590,
"price" : 31.06,
"format" : "paperback",
"rating" : 4.4,
"release_year" : "2017",
"vector_recommendation" : [
5.9,
4.4,
6.8
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "cryptonomicon",
"_score" : 1.7700485,
"_source" : {
"isbn13" : "978-1847249616",
"author" : "Neal Stephenson",
"title" : "Cryptonomicon",
"publisher" : "Avon",
"category" : [
"thriller",
"novel"
],
"pages" : 1152,
"price" : 6.99,
"format" : "paperback",
"rating" : 4.0,
"release_year" : "2002",
"vector_recommendation" : [
10.0,
4.0,
9.3
]
}
},
{
"_index" : "books",
"_type" : "_doc",
"_id" : "garbage-collection-handbook",
"_score" : 1.528916,
"_source" : {
"isbn13" : "978-1420082791",
"author" : [
"Richard Jones",
"Antony Hosking",
"Eliot Moss"
],
"title" : "The Garbage Collection Handbook: The Art of Automatic Memory Management",
"publisher" : "Taylor & Francis",
"category" : [
"programming algorithms"
],
"pages" : 511,
"price" : 87.85,
"format" : "paperback",
"rating" : 5.0,
"release_year" : "2011",
"vector_recommendation" : [
5.1,
5.0,
1.3
]
}
}
]
}
}我们可以看出来 “Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity” 是最贴近的书。我们可以看一下它的recommendation_vector:
"vector_recommendation" : [ 4.0, 4.0, 9.2 ]这是所有的书里最贴近我们搜索要求的书了。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
评论
登录后参与评论
暂无评论
推荐阅读
![PHP使用swoole实现多线程爬虫[通俗易懂]](https://ask.qcloudimg.com/http-save/yehe-8223537/0a09351fc10795ebff8511824041267c.png)
推荐阅读
相关推荐
PHP使用swoole实现多线程爬虫[通俗易懂]
更多 >
腾讯云开发者
