Class Lucene95HnswVectorsFormat
java.lang.Object
org.apache.lucene.codecs.KnnVectorsFormat
org.apache.lucene.backward_codecs.lucene95.Lucene95HnswVectorsFormat
- All Implemented Interfaces:
NamedSPILoader.NamedSPI
Lucene 9.5 vector format, which encodes numeric vector values and an optional associated graph
connecting the documents having values. The graph is used to power HNSW search. The format
consists of three files:
.vec (vector data) file
For each field:
- Vector data ordered by field, document ordinal, and vector dimension. When the vectorEncoding is BYTE, each sample is stored as a single byte. When it is FLOAT32, each sample is stored as an IEEE float in little-endian byte order.
- DocIds encoded by
IndexedDISI.writeBitSet(DocIdSetIterator, IndexOutput, byte)
, note that only in sparse case - OrdToDoc was encoded by
DirectMonotonicWriter
, note that only in sparse case
.vex (vector index)
Stores graphs connecting the documents for each field organized as a list of nodes' neighbours as following:
- For each level:
- For each node:
- [vint] the number of neighbor nodes
- array[vint] the delta encoded neighbor ordinals
- For each node:
- After all levels are encoded memory offsets for each node's neighbor nodes encoded by
DirectMonotonicWriter
are appended to the end of the file.
.vem (vector metadata) file
For each field:
- [int32] field number
- [int32] vector similarity function ordinal
- [vlong] offset to this field's vectors in the .vec file
- [vlong] length of this field's vectors, in bytes
- [vlong] offset to this field's index in the .vex file
- [vlong] length of this field's index data, in bytes
- [vint] dimension of this field's vectors
- [int] the number of documents having values for this field
- [int8] if equals to -1, dense – all documents have values for a field. If equals to 0, sparse – some documents missing values.
- DocIds were encoded by
IndexedDISI.writeBitSet(DocIdSetIterator, IndexOutput, byte)
- OrdToDoc was encoded by
DirectMonotonicWriter
, note that only in sparse case - [vint] the maximum number of connections (neighbours) that each node can have
- [vint] number of levels in the graph
- Graph nodes by level. For each level
- [vint] the number of nodes on this level
- array[vint] for levels greater than 0 list of nodes on this level, stored as the level 0th delta encoded nodes' ordinals.
-
Field Summary
FieldsModifier and TypeFieldDescription(package private) final int
The number of candidate neighbors to track while searching the graph for each newly inserted node.static final int
Default number of the size of the queue maintained while searching during a graph construction.static final int
Default number of maximum connections per node(package private) static final int
(package private) final int
Controls how many of the nearest neighbor candidates are connected to the new node.private static final int
The maximum size of the queue to maintain while searching during graph construction This maximum value preserves the ratio of the DEFAULT_BEAM_WIDTH/DEFAULT_MAX_CONN i.e.private static final int
A maximum configurable maximum max conn.(package private) static final String
(package private) static final String
(package private) static final String
(package private) static final String
(package private) static final String
(package private) static final String
(package private) static final int
(package private) static final int
Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat
DEFAULT_MAX_DIMENSIONS, EMPTY
-
Constructor Summary
ConstructorsConstructorDescriptionConstructs a format using default graph construction parametersLucene95HnswVectorsFormat
(int maxConn, int beamWidth) Constructs a format using the given graph construction parameters. -
Method Summary
Modifier and TypeMethodDescriptionfieldsReader
(SegmentReadState state) Returns aKnnVectorsReader
to read the vectors from the index.fieldsWriter
(SegmentWriteState state) Returns aKnnVectorsWriter
to write the vectors to the index.int
getMaxDimensions
(String fieldName) Returns the maximum number of vector dimensions supported by this codec for the given field nametoString()
Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat
availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat
-
Field Details
-
META_CODEC_NAME
- See Also:
-
VECTOR_DATA_CODEC_NAME
- See Also:
-
VECTOR_INDEX_CODEC_NAME
- See Also:
-
META_EXTENSION
- See Also:
-
VECTOR_DATA_EXTENSION
- See Also:
-
VECTOR_INDEX_EXTENSION
- See Also:
-
VERSION_START
static final int VERSION_START- See Also:
-
VERSION_CURRENT
static final int VERSION_CURRENT- See Also:
-
MAXIMUM_MAX_CONN
private static final int MAXIMUM_MAX_CONNA maximum configurable maximum max conn.NOTE: We eagerly populate `float[MAX_CONN*2]` and `int[MAX_CONN*2]`, so exceptionally large numbers here will use an inordinate amount of heap
- See Also:
-
DEFAULT_MAX_CONN
public static final int DEFAULT_MAX_CONNDefault number of maximum connections per node- See Also:
-
MAXIMUM_BEAM_WIDTH
private static final int MAXIMUM_BEAM_WIDTHThe maximum size of the queue to maintain while searching during graph construction This maximum value preserves the ratio of the DEFAULT_BEAM_WIDTH/DEFAULT_MAX_CONN i.e. `6.25 * 16 = 3200`- See Also:
-
DEFAULT_BEAM_WIDTH
public static final int DEFAULT_BEAM_WIDTHDefault number of the size of the queue maintained while searching during a graph construction.- See Also:
-
DIRECT_MONOTONIC_BLOCK_SHIFT
static final int DIRECT_MONOTONIC_BLOCK_SHIFT- See Also:
-
maxConn
final int maxConnControls how many of the nearest neighbor candidates are connected to the new node. Defaults toDEFAULT_MAX_CONN
. SeeHnswGraph
for more details. -
beamWidth
final int beamWidthThe number of candidate neighbors to track while searching the graph for each newly inserted node. Defaults toDEFAULT_BEAM_WIDTH
. SeeHnswGraph
for details.
-
-
Constructor Details
-
Lucene95HnswVectorsFormat
public Lucene95HnswVectorsFormat()Constructs a format using default graph construction parameters -
Lucene95HnswVectorsFormat
public Lucene95HnswVectorsFormat(int maxConn, int beamWidth) Constructs a format using the given graph construction parameters.- Parameters:
maxConn
- the maximum number of connections to a node in the HNSW graphbeamWidth
- the size of the queue maintained during graph construction.
-
-
Method Details
-
fieldsWriter
Description copied from class:KnnVectorsFormat
Returns aKnnVectorsWriter
to write the vectors to the index.- Specified by:
fieldsWriter
in classKnnVectorsFormat
- Throws:
IOException
-
fieldsReader
Description copied from class:KnnVectorsFormat
Returns aKnnVectorsReader
to read the vectors from the index.- Specified by:
fieldsReader
in classKnnVectorsFormat
- Throws:
IOException
-
getMaxDimensions
Description copied from class:KnnVectorsFormat
Returns the maximum number of vector dimensions supported by this codec for the given field nameCodecs implement this method to specify the maximum number of dimensions they support.
- Specified by:
getMaxDimensions
in classKnnVectorsFormat
- Parameters:
fieldName
- the field name- Returns:
- the maximum number of vector dimensions.
-
toString
-