Feature Extraction

The Feature Extraction detector uses a HuggingFace sentence-transformer to embed text content into dense vectors. The embeddings are stored as finding metadata and can be consumed by downstream semantic search pipelines, clustering jobs, or vector databases.

When to use

Semantic search — index document embeddings in a vector DB so users can query by meaning rather than exact keywords.
Duplicate detection — cluster embeddings to surface near-duplicate documents across sources.
Anomaly detection — identify documents that are semantically outliers within a collection.
Retrieval-augmented generation (RAG) — pre-compute embeddings for use in RAG pipelines; Classifyre handles the extraction, your pipeline handles retrieval.
Semantic similarity scoring — compare embeddings of new content against a known baseline to flag drift or unexpected topics.

How it works

At scan time, Classifyre loads the specified sentence-transformer model and embeds each asset’s text content. The pooling strategy (mean, cls, max) controls how per-token hidden states are aggregated into a single vector. L2 normalisation is applied by default, making cosine similarity equivalent to dot product.

Long documents are split into chunks when chunk_size is set. Each chunk produces its own embedding finding, allowing partial-document retrieval.

Configuration

Parameter	Type	Required	Description	Default	Constraints
model	string	Yes	HuggingFace hub ID (e.g. 'BAAI/bge-base-en-v1.5', 'sentence-transformers/all-MiniLM-L6-v2') or absolute local directory path.	—	—
model_revision	string	No	Git branch, tag, or commit hash when fetching from the HuggingFace hub.	null	—
device	string	No	Inference device: 'cpu' (default), 'cuda', 'mps', or a CUDA device string like 'cuda:0'.	cpu	—
pooling_strategy	string	No	How to aggregate per-token hidden states into a single embedding vector. Allowed: mean, cls, max, none	mean	—
normalize_embeddings	boolean	No	L2-normalise the final embedding vector. Recommended for cosine-similarity workloads.	true	—
truncation	boolean	No	Truncate input to the model's maximum sequence length.	true	—
max_length	integer	No	Override the tokenizer's default maximum sequence length.	null	—
batch_size	integer	No	Number of texts to encode in a single forward pass.	8	—
chunk_size	integer	No	Split text into chunks of this many characters before embedding. Each chunk produces its own finding.	null	—
chunk_overlap	integer	No	Character overlap between consecutive chunks.	0	—

Examples

General-purpose English embeddings

{
  "type": "FEATURE_EXTRACTION",
  "model": "BAAI/bge-base-en-v1.5",
  "pooling_strategy": "mean",
  "normalize_embeddings": true,
  "batch_size": 16
}

Multilingual embeddings with chunking

{
  "type": "FEATURE_EXTRACTION",
  "model": "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",
  "device": "cpu",
  "pooling_strategy": "mean",
  "normalize_embeddings": true,
  "chunk_size": 256,
  "chunk_overlap": 32,
  "batch_size": 8
}