Everyone needs an ISP
You can't get anywhere on the Internet without a way to connect to the Internet. By definition, your Internet Service Provider (ISP) is how you connect.
Think of the Internet has a series of roads and highways that are all interconnected and the site you want to access as your travel destination. Regardless of where you start, you begin by traversing a portion of that network of roads. Whether it's the road the runs in front of your flat, a driveway coming up to your house, or a 8-lane highway you climb the fence to get to, that first section of road is your ISP.
If you took your laptop to the nearest university and plugged directly into their cable feed coming from the Internet, they'd be your ISP--at least until you got caught.
Use of the Public Internet requires routers
Routers move traffic from one network to another. A network in this sense is a group of nodes on the same IP subnetwork. The Internet is nothing more than many individual IP subnetworks linked together. If all of the computers and servers in the world were on one giant IP subnetwork, routers would not be needed. But this is not how the Internet was designed (and for good reason). As a result, if you want to access a site that's hosted elsewhere on the Internet, your traffic must be "routed" (by routers) to its destination.
That said, it is perfectly possible to not have a router of your own and plug your computer directly into your ISP's network (assuming they let you do this). In that case, your computer would get an address valid on your ISP's network. However, this only eliminates one router from the equation. When your computer begins a conversation with that remote site you want to access, it will send all of the data packets to the ISP's router. Use of routers is entirely unavoidable on the public Internet.
Bypassing your ISP for DNS
You are correct that you don't need DNS to access (some) sites as long as you know its IP address. But you seem to be assuming you need to use your ISP for DNS, which is not the case. DNS is a public service and regardless what nameserver you use to resolve your lookup queries, you'll be able to get to the site you're trying to reach. For example, Google offers public DNS services at the IP addresses of 8.8.8.8 and 8.8.4.4.. If you configure your system to use those DNS servers, no queries are sent to your ISP.
Or doing away with DNS entirely
Perhaps you're wondering if you can dispense with DNS entirely. In practical terms the answer is no. One immediate problem you'd face is the fact a server that host multiple websites looks at the name you type into your browser's address bar to figure out which website it should show you. While you could work around this problem using your HOSTS file, there are other DNS record types that can only be accessed through a DNS query.