As the documentation page says, it depends on which form of the function you use:
[X,map] = rgb2ind(RGB,n): if you specify the number of colors as input, this will use minimum variance quantization to build an indexed image with at most n colors
[X,map] = rgb2ind(RGB,tol): if you specify a tolerance value as input, it uses uniform quantization to build indexed image with at most (floor(1/tol)+1)^3 colors
X = rgb2ind(RGB,map): if you specify a colormap as input, it will use the inverse colormap algorithm to build indexed image mapped to the specified map
You could always read the source code yourself (edit rgb2ind)
Here are examples showing how to use all forms of the functions:
%% some truecolor image
RGB = imread('pears.png');
imshow(RGB)
%% 16 colors
[X,map] = rgb2ind(RGB, 16);
imshow(X,map)
%% 0.15 tolerance, no dithering
[X,map] = rgb2ind(RGB, 0.15, 'nodither');
imshow(X,map)
%% use a pinkish colormap with 32 colors
map = pink(32);
X = rgb2ind(RGB, map);
imshow(X,map)