Why the buffer size is equal 70_000?
Why buffer size > samples in train data:70_000>60_000? and what's difference between :
buffer size > samples , buffer size < samples and buffer size = samples??
- Understanding the Shuffle Buffer in TensorFlow
Let’s assume the MNIST dataset contains 1 million samples, and we set the buffer size to 70,000.
This means TensorFlow will:
Load the first 70,000 samples into a temporary buffer.
Randomly shuffle the samples within this buffer.
As each sample is taken out for training, a new sample from the remaining dataset is added to the buffer.
This process continues until all 1 million samples have been seen.
As a result, we end up with the entire dataset approximately randomized.
We can then use this shuffled dataset to extract training and validation sets.
- Choosing the Buffer Size
Setting the buffer size (e.g., 70,000 or 60,000) is up to you — the choice depends mainly on your machine’s memory capacity.
A larger buffer size gives better randomness (closer to a perfect shuffle).
However, a larger buffer also requires more memory.
A smaller buffer size still works but produces slightly less random shuffling.
- Summary
It doesn’t matter if your buffer size is greater than, less than, or equal to the total number of samples (e.g., 60,000 for MNIST).
The key idea is:
The larger the buffer size, the better the shuffling — but it’s limited by your system’s available memory.