Manticore Search version 29.9.0 introduces an 'Auto-chunking' feature, allowing AI to search long documents more accurately and efficiently.
Imagine this: You give AI a massive 50-page technical report and ask, “Summarize the three most important takeaways from this document.” You’d be pretty frustrated if the AI only scanned the beginning and replied, “The document was too long, so I couldn’t read it all,” missing the crucial conclusion.
In fact, this happens often when talking to AI. It’s due to the ‘token limit’ (the capacity for word chunks) that AI has. However, a technology that solves this problem very simply has recently emerged.
Why It Matters
In AI services we use, especially in ‘Retrieval-Augmented Generation’ (RAG), an AI system that analyzes large volumes of documents, ‘accuracy’ of information is vital. But with existing methods, when long documents spanning hundreds of pages are fed to an AI, the AI often cannot even read or ends up discarding parts that exceed the established token limit.
The new feature introduced in the database engine Manticore Search fundamentally prevents this ‘data loss.’ As AI becomes capable of finding information more intelligently, work efficiency will increase, and the reliability of AI assistants will be significantly improved.
The Explainer
Shall we use an analogy?
If you give a child an entire giant encyclopedia at once and tell them to “find the information,” versus giving them the encyclopedia divided into small chapters by topic and telling them to “find the information”—which would be faster and more accurate? Naturally, it would be the latter.
Until now, AI search often tried to read the entire encyclopedia at once, got exhausted, and skipped the end. Manticore Search’s ‘Auto-chunking’ is a technology that automatically cuts documents into a size perfect for the AI to read at once when they are saved to the database. Source: Manticore Search 29.9.0 adds auto-chunking for long documents
Simply put, it means breaking down massive documents into ‘small portions’ that are easy for AI to understand. By doing this, even long documents can go through the AI’s ‘embedding’ (a technology that converts the meaning of text into numbers so the AI can understand it) process without anything left out. Source: Better Vector Search for Long Documents: Chunking Inside …
Where We Stand
This feature is officially supported starting from Manticore Search version 29.9.0. Previously, developers had to manually build complex, separate splitter libraries or data processing pipelines, but now this can be solved easily just through database configuration. Source: Better Vector Search for Long Documents: Chunking Inside …
In particular, internal tests conducted by Manticore Search showed that search recall for long documents improved dramatically from the existing 55% to 83%. Source: Better Vector Search for Long Documents: Chunking Inside …
Using it is also very simple. When creating a database table, you only need to include a configuration value called chunk_strategy in the vector column. Now, instead of one document being compressed into a single representative numerical value (vector), it can have multiple vectors, enabling much more detailed information retrieval. Source: Manticore Search Adds Built-In Document Chunking to Improve … Source: Manticore Search 29.9.0: Chunked auto-embeddings and mmap …
What’s Next
As AI becomes capable of handling longer information, companies will be able to more effectively train and utilize vast knowledge-based documents for AI. Furthermore, such ‘pre-processing at the database level’ features will become increasingly popularized. This is expected to greatly reduce the inconvenience of developers having to write complex code themselves every time to overcome the limitations of AI models.
MindTickleBytes AI Reporter’s View
To those who felt frustrated watching AI fail to read long documents and provide sloppy summaries, this update clearly shows that ‘how information is delivered to AI’ is just as important as ‘AI getting smarter.’ It will be interesting to see how this technical evolution, where the database assists the AI’s brain, makes future RAG systems even smarter.
References
- Automatic language translation
- Auto-chunking
- Real-time video generation
- Documents are deleted too quickly
- The latter part of the document is arbitrarily omitted
- Language detection fails
- chunk_strategy
- document_splitter
- long_doc_mode