The engine is currently under heavy development and the documentation may be slightly out of date. As the API stabilizes, both the content and scope of the present documentation will increase. For any inquiries, bugs or feature requests, be sure to contact the developers or file an issue.
Introduction
Garamond is a small, flexible neural and data search engine. Search functionality available through API method calls from Julia code or through communication with a Garamond search server.
Installation
The Garamond repository can be downloaded through git:
$ git clone https://github.com/zgornel/Garamond.jlor from inside Julia. Entering the Pkg mode with ] and writing:
add https://github.com/zgornel/Garamond.jl#masterdownloads the master branch of the repository and adds Garamond to the current active environment.
Architecture
Architecturally, the engine employs an analytical database and a series of search indexes to perform the search. The database acts as data source to create the indexes (searcher objects), for building responses for clients and as input source for the ranking and recommendation operations. Incoming search/recommendation/ranking requests are parsed to an internal request format and sent to the database and searchers, depending on the request. Each searcher has its own characteristics i.e. ways of embedding documents, searching through the vectors. Search results from searchers can be combined in a variety of ways. The results are then filtered, aggregated and a response is built and sent to the client. Any number of searcher objects can be created and each of these can re-use data embedders i.e. objects that transform data into numerical vectors. Data can be either indexed while the engine is running online i.e. pushed into all indexes plus database or, by fully reloading and reindexing. The engine supports runtime compilation of custom data loaders, parsers, rankers and recommenders. These can be thin clients that connect i.e. through TCP sockets, HTTP, to external systems that perform the actual workload, provided that the internal client APIs are implemented.
Main features
- In-memory analytical db based on JuliaDB
- Pluggable support for custom parsers, loaders, recommenders and rankers
- Millon-scale indexing using hnsw
- Billion-scale search through IVFADC
- Online indexing i.e. fast push/pop to/from index and db
- Runtime batch re-indexing i.e. reindex while operating
- HTTP(REST), Web-socket and UNIX socket connectivity
- Complex query search patterns supported
- Data embedders are a shared resource - each searcher can have different input and data embedders
- Wordvectors support: Word2Vec, ConceptnetNumberbatch, GloVe
- Compressed vector support for low-memory footprint using array quantization
- Classic search based on term frequency, tf-idf, bm25
- Suggestion support using BK Trees
- Many state-of-the-art neural document and sentence embedding methods
- Multi-threading supported
- Caching mechanisms for fast resume
- Portable and statically compilable to many architectures
Longer term plans
- Extend search to Images, Videos and Audio i.e. generic search
- Peer-to-peer / distributed operations support