five types of chunking in rag-

1. Fixed size chunking
2. Semantic chunking
3. Recursive chunking
4. Structural chunking
5. LLM chunking

fixed size-

chunk(200mb)-hyperparameter

divide pages into chunks of fixed size

advantages- quick, fast processing

disadvantages- semantic breaks, lost context

Semantic chunking-

(threshold=0.8)-hyperparameter

tries to solve problem of fixed size chunking(no meaning of chunks connections)

• it first convert s1 and s2 into vector embedings
• then checks the cosine similarity of s1 with s2 ,if greater than certain threshold it add the s2 into chunk1
• similarly for s3, s4 create vector embedding and check similarity if greater then threshold add them to chunk1