Weaviate

Features

Feature	Supported?(Yes/No)	Notes
Full Refresh Sync	No
Incremental - Append Sync	Yes
Incremental - Append + Deduped	No
Namespaces	No
Provide vector	Yes

Output Schema

Each stream will be output into its own class in Weaviate. The record fields will be stored as fields in the Weaviate class.

Uploading Vectors: Use the vectors configuration if you want to upload vectors from a source database into Weaviate. You can do this by specifying the stream name and vector field name in the following format:

<stream_name>.<vector_field_name>, <stream_name2>.<vector_field_name>

For example, if you have a table named my_table and the vector is stored using the column vector then you should use the following vectorsconfiguration: my_table.vector.

Dynamic Schema: Weaviate will automatically create a schema for the stream if no class was defined unless you have disabled the Dynamic Schema feature in Weaviate. You can also create the class in Weaviate in advance if you need more control over the schema in Weaviate.

IDs: If your source table has an int based id stored as field name id then the ID will automatically be converted to a UUID. Weaviate only supports the ID to be a UUID. For example, if the record has id=1 then this would become a uuid of 00000000-0000-0000-0000-000000000001.

Any field name starting with an upper case letter will be converted to lower case. For example, if you have a field name USD then that field will become uSD. This is due to a limitation in Weaviate, see this issue in Weaviate.

Getting Started

Airbyte Cloud only supports connecting to your Weaviate Instance instance with TLS encryption and with a username and password.

Getting Started (Airbyte Open-Source)

Requirements

To use the Weaviate destination, you'll need:

• A Weaviate cluster version 21.8.10.19 or above

Configure Network Access

Make sure your Weaviate database can be accessed by Airbyte. If your database is within a VPC, you may need to allow access from the IP you're using to expose Airbyte.

Permissions

You need a Weaviate user or use a Weaviate instance that's accessible to all

Setup the Weaviate Destination in Airbyte

You should now have all the requirements needed to configure Weaviate as a destination in the UI. You'll need the following information to configure the Weaviate destination:

• URL for example http://localhost:8080 or https://my-wcs.semi.network
• Username (Optional)
• Password (Optional)
• Batch Size (Optional, defaults to 100)
• Vectors a comma separated list of <stream_name.vector_field_name> to specify the field
• ID Schema a comma separated list of <stream_name.id_field_name> to specify the field name that contains the ID of a record

Changelog

Version	Date	Pull Request	Subject
0.1.1	2022-02-08	#22527	Multiple bug fixes: Support String based IDs, arrays of uknown type and additionalProperties of type object and array of objects
0.1.0	2022-12-06	#20094	Add Weaviate destination