As a starter, a few notes about your original query:
it is not valid SQL; you have an aggregate function in the select clause (max(t.hour)), but no group by clause
the order by clause of rank() is not optimized: it starts with columns that are part of the partition by clause, which is unecessary (we already know that all rows in the partition have the same values for these columns)
So I'll assume that the query you want to rewrite is:
select
t.storeid as storeid,
t.artid as artid,
t.totamount,
t.hour,
rank() over(partition by t.storeid, t.artid order by t.hour desc) rn
from t
order by t.storeid, t.artid
Now, in MySQL 5.7, one method to emulate rank() uses a correlated subquery:
select
t.storeid as storeid,
t.artid as artid,
t.totamount,
t.hour,
(
select 1 + count(*)
from t t1
where
t1.storeid = t.storeid
and t1.artid = t.artid
and t1.hour > t.hour
) as rn
from t
order by t.storeid, t.artid
Note that this is really less efficient that window functions, since the subquery must be executed for each and every row in the original table. Other solutions typically involve user variables.