You can try doing the following. Here the vector unique_years contains the years in your datetime vector and newdatemat is a cell array with one row for every group of dates associated with a given year.
%// Sample data
datemat=[datetime(2015,5,3);...
         datetime(2015,3,5);...
         datetime(2016,4,2);...
         datetime(2014,1,1);...
         datetime(2014,3,2);...
         datetime(2017,3,3)];
%// Get the unique years from the date vector
unique_years = unique(year(datemat));
%// create a cell array newdatemat that contains
%// the datenums from datemat grouped by year
newdatemat=cell(length(unique_years), 1);
for ii=1:length(unique_years)
    newdatemat{ii} = datemat(year(datemat) == unique_years(ii));
end
An analogous version of the code above is provided below, following the suggestion of @Adriaan:
mydatevec = datevec(datemat);
unique_years = unique(mydatevec(:,1));
newdatemat1=cell(length(unique_years), 1);
for ii=1:length(unique_years)
    newdatemat1{ii} = mydatevec(mydatevec(:,1) == unique_years(ii),:);
end