To address your two things in order:
The libtool script intercepts that install instruction and also installs the .so file in the appropriate place.
It's putting it in /usr/local/lib probably because you listed it in lib_LTLIBRARIES (although I can't be sure if you don't show your code) and your --prefix is set to its default of /usr/local.
This last one is difficult since Autotools' official stance is that all user-installed programs belong in /usr, whereas many other tools expect things to be in /lib/something. Here's one way to do it, that I personally consider wrong:
# Don't do this
libsecuritydir = /lib/security
libsecurity_LTLIBRARIES = pam_mymodule.la
This bypasses --prefix, which will go horribly, horribly wrong if you try to do a local install of your package without writing directly into your live system, which, trust me, you will want to do at some point. It will also prevent you from packaging your program in most Linux distributions' packaging systems.
The correct way is to push the responsibility onto whoever installs the package: add a --with-libsecuritydir argument to configure.ac using AC_ARG_WITH and let that default to $(libdir)/security:
AC_ARG_WITH([libsecuritydir],
[AS_HELP_STRING([--with-libsecuritydir],
[Directory for PAM modules, pass /lib/security for live install])],
[], [with_libsecuritydir='$(libdir)/security'])
AC_SUBST([libsecuritydir], [$with_libsecuritydir])
and then just do
libsecurity_LTLIBRARIES = pam_mymodule.la
in Makefile.am.
When you want to install a live version directly into your system (or are building a binary package) pass --with-libsecuritydir=/lib/security to configure.