It's important to understand the principal nature of these five different kinds of data / symbol:
1. 'my_tbl'
A string literal of unknown type. When used in SQL (embedded in plpgsql code or not), it is coerced to a type derived from the context. If the type cannot be determined an explicit cast may be required. Like: 'my_tbl'::text.
2. 'my_tbl'::text
The same string literal cast to type text. It can hold the name of a table, but it's really just text.
3. 'my_tbl'::regclass
An object identifier (OID) for a registered class. It is displayed and can be input as string representing a valid object name ('my_tbl'). The output is automatically schema-qualified ('my_schema.my_tbl') and / or double-quoted ('"mY_TbL"') if it would be ambiguous or illegal otherwise. It can be a regular table, sequence, view, materialized view, composite type etc. Details in this related answer:
4. my_tbl_var my_tbl (short for my_tbl_var my_tbl%ROWTYPE)
In the DECLARE section of a plpgsql code block that's a variable declaration with a well known row type (a.k.a. composite type). The type has to be registered in the system table pg_class (same as with a regclass variable). It's not the OID of the referenced object, but its actual row type. my_tbl_var and my_tbl are both identifiers here and cannot be parameterized. You can also cast any row or record directly: (123, 'foo')::my_tbl
5. my_tbl_var record
In the DECLARE section of a plpgsql code block that's the declaration of an anonymous record. Basically, a placeholder for a yet unknown row type / with yet undefined structure. It can be used in most of the places a row type can be used. But you cannot access fields from it before the record variable is assigned.
You were confusing 1., 3. and 4. and solved it by using 5. instead.
But there is more going wrong here:
- You are selecting a whole table, but a row (record) variable can only hold one row at a time. So only the first is assigned and returned. While there is no - ORDER BYclause, the result is arbitrary and can change any time. Evil trap.
 
- Since you are now using a - recordtype, you need to make sure it has been assigned before you can run tests on its fields, or you'll get exceptions for empty tables.
In your case the check- record_var IS NULLalmost does the same job. But there is a corner case for rows with NULL in all fields: then- record_var IS NULLevaluates to true. Even trickier for the test- IS NOT NULL. Details here:
 - I added a demo to the SQL fiddle below. 
- The function returns a single scalar (- boolean) value. Use:
 - RETURN false;
 - Instead of: - RETURN QUERY SELECT false;
 
Function
CREATE FUNCTION check_valid(_tbl regclass)
  RETURNS bool AS
$func$
DECLARE
   r record;
   _row_ct int;
BEGIN
   EXECUTE '
   SELECT is_valid, hit_count, hit_limit
   FROM  ' || _tbl || '
   ORDER  <whatever>
   LIMIT  1'            -- replace <whatever> with your sort criteria
   INTO r;              -- only needed columns
   GET DIAGNOSTICS _row_ct = ROW_COUNT;
   IF _row_ct = 0 THEN  -- necessary, because r may not be assigned
      RETURN false;
   ELSIF NOT r.is_valid OR r.hit_count > r.hit_limit THEN
      RETURN false;
   END IF;
   RETURN true;
END
$func$  LANGUAGE plpgsql;
SQL Fiddle (with two variants of the function and a demo for row IS NULL).
Major points
- Use - GET DIAGNOSTICSto find out whether any rows were found in a dynamic statement with- EXECUTE.
 
- The - IFexpression can be simplified.
 
- The parameter is of type - regclass, not just a tablename. I wouldn't use the misleading name "tablename" for this parameter. That only adds to your initial confusion. Calling it- _tblinstead.
 
If you'd want to also return a set of variable row type: