The use case you have is already well defined in the Material theme spec.
Here I am going to explain how to style color of any widget (eg: button/ text/ customWidget..) depending on the parent widget color. 
example use cases:
- primary colored text on white colored appBar
 
- white colored text on primary colored appBar
 
- black color text on white color surface
 
- primary color text on white color surface
 
Here you go. 
Step 1:
First of all, 
Define a primary color swatch that define all the variants from 50 to 900. 
final MaterialColor lightPrimaryColorSwatch = MaterialColor(
    0xff4f9af0,
    {
      50: Color(0xffeaf3fd),
      100: Color(0xffcae1fb),
      200: Color(0xffa7cdf8),
      300: Color(0xff84b8f5),
      400: Color(0xff69a9f2),
      500: Color(0xff4f9af0),
      600: Color(0xff4892ee),
      700: Color(0xff3f88ec),
      800: Color(0xff367ee9),
      900: Color(0xff266ce5),
    },
  );
Step 2:
You need to define a ColorScheme object that define all the 12 colors of material spec.
final ColorScheme lightColorScheme = ColorScheme(
        brightness: Brightness.light,
        primary: Color(0xff4f9af0),
        primaryVariant: Color(0xff2c86ed),
        secondary: Color(0xff0863c4),
        secondaryVariant: Color(0xff259b24),
        error: Color(0xffb00020),
        background: Color(0xfff7f8fa),
        onError: Colors.white,
        onSecondary: Colors.white,
        onBackground: Color(0xff292929),
        onPrimary: Colors.white,
        onSurface: Color(0xff292929),
        surface: Colors.white,
      );
Step 3:
Define theme data for your MaterialApp using the above defined values.
 return MaterialApp(
      theme: ThemeData(
        brightness: Brightness.light,
        primarySwatch: lightPrimaryColorSwatch,
        primaryColor: lightColorScheme.primary,
        primaryColorDark: lightColorScheme.primaryVariant,
        errorColor: lightColorScheme.error,
        colorScheme: lightColorScheme,
        primaryColorBrightness: Brightness.dark,
        accentColorBrightness: Brightness.dark,
        accentColor: lightColorScheme.secondary,
        primaryColorLight: lightColorScheme.secondaryVariant,
        backgroundColor: lightColorScheme.background,
        canvasColor: lightColorScheme.background,
      ),
    //... other attributes go here
  );
Now you are all set. From now on, you can use those 12 colors for any type of combo that you wish. 
Here is an example. 
import 'package:flutter/material.dart';
void main() {
  runApp(MyApp());
}
class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final MaterialColor lightPrimaryColorSwatch = MaterialColor(
      0xff4f9af0,
      {
        50: Color(0xffeaf3fd),
        100: Color(0xffcae1fb),
        200: Color(0xffa7cdf8),
        300: Color(0xff84b8f5),
        400: Color(0xff69a9f2),
        500: Color(0xff4f9af0),
        600: Color(0xff4892ee),
        700: Color(0xff3f88ec),
        800: Color(0xff367ee9),
        900: Color(0xff266ce5),
      },
    );
    final ColorScheme lightColorScheme = ColorScheme(
      brightness: Brightness.light,
      primary: Color(0xff4f9af0),
      primaryVariant: Color(0xff2c86ed),
      secondary: Color(0xff0863c4),
      secondaryVariant: Color(0xff259b24),
      error: Color(0xffb00020),
      background: Color(0xfff7f8fa),
      onError: Colors.white,
      onSecondary: Colors.white,
      onBackground: Color(0xff292929),
      onPrimary: Colors.white,
      onSurface: Color(0xff292929),
      surface: Colors.white,
    );
    return MaterialApp(
      theme: ThemeData(
        brightness: Brightness.light,
        primarySwatch: lightPrimaryColorSwatch,
        primaryColor: lightColorScheme.primary,
        primaryColorDark: lightColorScheme.primaryVariant,
        errorColor: lightColorScheme.error,
        colorScheme: lightColorScheme,
        primaryColorBrightness: Brightness.dark,
        accentColorBrightness: Brightness.dark,
        accentColor: lightColorScheme.secondary,
        primaryColorLight: lightColorScheme.secondaryVariant,
        backgroundColor: lightColorScheme.background,
        canvasColor: lightColorScheme.background,
      ),
      debugShowCheckedModeBanner: false,
      home: Scaffold(
        body: Center(
          child: MyWidget(),
        ),
      ),
    );
  }
}
    class MyWidget extends StatelessWidget {
      @override
      Widget build(BuildContext context) {
        return Column(
          children: <Widget>[
            SizedBox(height: 8.0),
           ///white color text on container with primary color
            Container(
                color: Theme.of(context).colorScheme.primary,
                height: 40.0,
                width: 100.0,
                alignment: Alignment.center,
                child: Padding(
                  padding: EdgeInsets.all(8.0),
                  child: Text('Hello, World!',
                      style: Theme.of(context).textTheme.bodyText2.copyWith(
                          color: Theme.of(context).colorScheme.onPrimary)),
                )),
            SizedBox(height: 8.0),
            ///Primary color text on  white color container.
            Container(
                color: Theme.of(context).colorScheme.surface,
                height: 40.0,
                width: 100.0,
                alignment: Alignment.center,
                child: Padding(
                  padding: EdgeInsets.all(8.0),
                  child: Text('Hello, World!',
                      style: Theme.of(context)
                          .textTheme
                          .bodyText2
                          .copyWith(color: Theme.of(context).colorScheme.primary)),
                )),
          ],
        );
      }
    }
You can find a live demo here.